From natural to artificial cognitive systems
Transcript of From natural to artificial cognitive systems
![Page 1: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/1.jpg)
From natural toto
artificial cognitive systems
Giorgio MettagLIRA-Lab
Dept. of Computer and Systems Sciences
University of Genoa Italy
Dept. of Robotics, Brain and Cognitive Sciences
Italian Institute of TechnologyGenoaUniversity of Genoa, Italy Genoa
![Page 2: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/2.jpg)
The Italian Institute of Technologyh ta an nst tut of chno ogy
![Page 3: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/3.jpg)
Robotics, Brain and Cognitive Sciences D t tDepartment
Contact PADsCNT-coated Pt electrodes Micromachined
substrates
CNT
Contact PADsCNT-coated Pt electrodes Micromachined
substrates
CNT
Exploring the brain mechanisms at the basis of motor behavior and motor learning
Human behavior
CNT-coated electrode
CNT-coated electrode
Brain machine interfaceHuman behavior
The development of new generation brain imaging systems:open magnet high resolution FMRI system targeted to brain research
and natural behavior analysis (manipulation)
Design and fabrication of neural probes for in-vivo applications bases on conductive and biocompatible nano-materials for electrodes
and natural behavior analysis (manipulation).
Brain machine interface
Humanoid robotics
Brain machine interface
Designing new generation, intelligent and interacting
Toward next generation hybrid systems realized with soft materials and with cognitive abilities allowing operating in human environmentHumanoid robotics
The near-infrared spectroscopy: a new non-invasive technique to investigate brain functions in human subjects
platforms
![Page 4: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/4.jpg)
The tale of the Wright brothers( d th )(and three messages)
1. Reverse engineering:– Looking and copying bird flight → aircraft design– Reverse of reverse engineering → aerodynamics led to
better understanding of bird flight (forward engineering)2 Models:2. Models:
– Wright started from a previous model of Lilienthal (which was wrong) → but then they had (after 2 years) to produce their own models (and test them), they built a wind tunnel ( ), y(very modern)!
3. Stability and control:• Separate models didn’t work well (either stability or
t l)control)• Discovered that the key to stability and control is by rolling
→ turn by rolling! Separate models don’t work, holistic approach is required and this was done by looking also at pp q y gbirds.
• Understanding at the systems level* From The Computational Neurobiology of Reaching and Pointing, Shadmehr and Wise, MIT press
![Page 5: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/5.jpg)
and the story goes…and the story goes
• Late 1903 first powered flight (35m • Late 1903, first powered flight (35m, 10km/hour)5 l t 2 h fli ht• 5 yrs later, 2 hours flight
• 8 yrs later, across North America• 24 yrs later, New York to Paris• 65 yrs later three people to the moon65 yrs later, three people to the moon• Now, small seats and screaming infants
![Page 6: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/6.jpg)
The three messages again!The three messages again!
1. Reverse and forward engineering
2. Mathematical modeling and empirical testingtesting
3. Systems-level approach
![Page 7: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/7.jpg)
Reverse engineeringReverse engineering
![Page 8: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/8.jpg)
Spinal behaviorsSpinal behaviors
Walking behavior: cat rehabilitated to walkafter complete spinal cord transection
Wiping reflex: an irritating stimuluselicits a wiping movement preciselydirected at the stimulus location
Poppele, R., & Bosco, G. (2003). Sophisticated spinal contributions to motor control. Trends in Neurosciences, 26(5), 269-276.
![Page 9: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/9.jpg)
Spinal sensorimotor coordinationp(not simple reflexes after all)
• Stimulus-response coordinate transformations– Evidence for:
• Combination of basic behaviors• Reach body parts that move respect to each other• Adapt and avoid obstacles• Use different sets of muscles
– Inverse dynamics (less compelling to me):• Move → dynamics• Descending “kinematics” pathways (e.g. optic tectum, orienting
behavior)• Walking, CPG’s, etc.
– Degrees of freedom problem:• Evidence of synergies• Muscles activate together• Multi-joint muscles
![Page 10: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/10.jpg)
Differences in theDifferences in the muscle activation
![Page 11: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/11.jpg)
Observed field features (2)Modeling the spinal controller: Francesco Nori
• Systematic stimulation of different regions of the spinal cord produced only a few types force fields (at least p y yp (four).
Dissimilarity matrix of 49 force fields. Darker circles representd(F F )<<1d(F1,F2)<<1
F49
4 clusters can beidentified4 clusters can beidentifiedidentifiedidentified
F1 F49
F1
The presence of only few units of motor output within the spinal cord is difficult to reconcile with the obvious ability of the nervous system to produce a wide range of movements.
![Page 12: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/12.jpg)
Looking at musclesLooking at muscles
H-reflex deviation from the mean(conductance of the spinal-muscle nerves)
Muscle anticipate the kinematics
kinematics
Borroni P Montagna M Cerri G & Baldissera F (2005) Cyclic time course
Prone position, wrist ext/flexion (ECR, FCR muscles)
Borroni, P., Montagna, M., Cerri, G., & Baldissera, F. (2005). Cyclic time course of motor excitability modulation during the observation of a cyclic hand movement. Brain Research, 1065, 115-124.
![Page 13: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/13.jpg)
Supine positionSupine position
As before, but differentphase difference btw thephase difference btw thekinematics and the muscularactivation
![Page 14: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/14.jpg)
Then compare with actual actionThen compare with actual action
Which was 54° on average EMG signals fromthe FCR and ECRmuscles
Which was 112° on average
The movement doesn’t change
![Page 15: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/15.jpg)
…but let me take a little leap forward
![Page 16: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/16.jpg)
The experimentThe experiment
• Listening: three categories of stimuli (words, pseudo-words, bi-tonal sounds).
• Two phonemes ‘rr’ requires strong tongue tip movement, ‘ff’ requires slight tongue tip m vementmovement.
• TMS of the under-threshold motor cortex.R di f th MEP ( t k d • Recording of the MEP (motor-evoked potential) from the tongue muscles.
![Page 17: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/17.jpg)
Examples ofd/ d dword/pseudo-words
![Page 18: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/18.jpg)
Results (in short)Results (in short)
Modulation due to meaningModulation due to meaning
Effect due to phoneme
0.0
![Page 19: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/19.jpg)
“In all communication, sender and receiver must be bound by a common understanding about what counts; gwhat counts for the sender mustcount for the receiver, else communication does not occur.communication does not occur.Moreover the processes of productionand perception must somehow be li k d th i t ti t t linked; their representation must, at some point, be the same.”[Alvin Liberman, 1993]
![Page 20: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/20.jpg)
More central
More peripheral
More central
Even more peripheral(multiple body parts)
![Page 21: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/21.jpg)
Grasping neuronsp g
![Page 22: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/22.jpg)
F5 canonical neuronsA B
1 s
20 sp/s
Observation + action Observation onlyObservation + action Observation only
![Page 23: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/23.jpg)
Mirror NeuronsMirror NeuronsThe neuron is activated by “seeing” someone else’s hand performing
i l ti ti d hil th k i f i tha manipulative action and while the monkey is performing the same action
From: Fadiga L L Fogassi V Gallese and G Rizzolatti Visuomotor Neurons: ambiguity of the
The type of action seen is relevant
From: Fadiga, L., L. Fogassi, V. Gallese, and G. Rizzolatti, Visuomotor Neurons: ambiguity of the discharge or "motor“ Perception? Internation Journal of Psychophysiology, 2000. 35: p. 165-177.
![Page 24: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/24.jpg)
ModelsModels
![Page 25: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/25.jpg)
Forwardmodel
Efferentcopy
Gripforce
controller
RealizedDesired
trajectory Inverse
Fingers Gripforce
controller
Arm Realizedtrajectory
j y Inversemodel
From: Flanagan JR, Wing AM. The role of internal models in motion g , gplanning and control: evidence from grip force adjustments during movements of hand-held loads. Journal of Neuroscience 1997, 17:1519-1528.
![Page 26: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/26.jpg)
Controlled
Stiffness, viscosity
Feedbackmotor
commandRealizedtrajectoryDesired
trajectory +Controlled
objectGain
-
Delay
Controlled
Feedforwardmotor
command
Realizedtrajectory
Desiredtrajectory
Inverse Controlledobject
Inversemodel
From: M. Kawato. Internal models for motor control and trajectory planning. j y p gCurrent Opinion in Neurobiology 1999, 9:718–727
![Page 27: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/27.jpg)
Effect of delaysEffect of delaysMake feedback control either poor or unstable altogether
• Engineering control systems
• Humans– Delays:
– Delays: 500μs– Movement duration:
seconds
• 20-50ms (spinal)• 150-250ms (vision)
– Movement duration: 150-– Gain of the controller:
can be made high500ms
– Gain of the controller: stiffness and muscle
• BTW: maintaining fast control loops is not an
f t
viscoelastic properties (comparatively low)
easy feat
![Page 28: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/28.jpg)
Building the internal modelsBuilding the internal models• Rote-learning would be unpractical:g p
– Too many possible actions (dof) for the available number of neurons (although they’re quite a lot!)
• Generalizing past experience:• Generalizing past experience:– Past experience is bound (unfortunately) to “represent”
only a portion of the whole state spaceD l i d di h l • Developing and extending the control structure to new behaviors– Sequencing and combining primitive behaviors q g g p
appropriately• Predicting the future course of action
It might turn out to be useful!– It might turn out to be useful!
![Page 29: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/29.jpg)
From: Lackner JR, Dizio P. Gravitoinertial force background level affects adaptation to Coriolis force perturbations of reaching movements.Journal of Neurophysiology 1998, 80:546-553.
![Page 30: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/30.jpg)
Internal models for interceptionInternal models for interception
Experimental setup
Stimuli (3 conditions)
With Alessandra Sciutti, Francesco Nori, Thierry Pozzo
![Page 31: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/31.jpg)
Some resultsSome results
Random
Experiment with motor information
![Page 32: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/32.jpg)
The wind tunnelThe wind tunnel
![Page 33: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/33.jpg)
The iCub: quick summaryq yThe iCub is the humanoid baby-robot being designed as part of the RobotCub project
– The iCub is a full humanoid robot sized as a three and half year-old child.f y
– The total height is 104cm.– It has 53 degrees of freedom, including
articulated hands to be used for manipulation articulated hands to be used for manipulation and gesturing.
– The robot will be (once the software is done) able to crawl and sit and autonomously able to crawl and s t and autonomously transition from crawling to sitting and vice-versa.
– The robot is GPL/FDL: software, hardware, , ,drawings, documentation, etc.
![Page 34: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/34.jpg)
Degrees of freedomDegrees of freedom• 53: 9 in each hand53 9 in each hand• Sensors: position, torque, temperature• And also: cameras microphones • And also: cameras, microphones,
gyroscopes, linear accelerometers• For the future: tactile sensors skin • For the future: tactile sensors, skin…
– Low-resolution:• Distributed many sensing points * Old versionDistributed many sensing points
– Fingertips:• Localized, high-resolutiong
![Page 35: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/35.jpg)
A few examplesA few examplesCustom design
Hand sensor sampling PIC-based card
Force/torque sensor fitted into the sensor
Design and documentation
Fingertip sensorization
Wired with 25micron coated wires
Design and documentation
![Page 36: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/36.jpg)
Current statusCurrent status
Design RealityDesign Reality
![Page 37: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/37.jpg)
Facial expressionsFacial expressions
![Page 38: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/38.jpg)
The soul of a new machineThe soul of a new machine
![Page 39: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/39.jpg)
More integrationMore integration
![Page 40: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/40.jpg)
and robot Yoga… and robot Yoga
![Page 41: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/41.jpg)
Skin, architecture by Marco MaggialiSkin, architecture by Marco Maggiali
CHIP
Sensor Element
Single element Flexible PCB
ArchitectureArchitecture
With Giorgio Cannata (Univ. of Genoa)
![Page 42: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/42.jpg)
Prototype d b ili bbcovered by silicon rubber
![Page 43: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/43.jpg)
Fingertip sensorsFingertip sensors
CAD/concept
Prototype Some testing Fingernail + microphone
By Alexander Schmitz Marco RandazzoBy Alexander Schmitz, Marco Randazzo, Marco Maggiali and Lorenzo Natale
![Page 44: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/44.jpg)
The handsThe hands
![Page 45: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/45.jpg)
Joint-level torque sensingJoint-level torque sensing
FEM analysis of deformation
Existing parts
By Alberto Parmiggiani and collaborating with Nikos
Changes (under implementation)Changes (under implementation)
By Alberto Parmiggiani and collaborating with Nikos Tsagarakis
![Page 46: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/46.jpg)
High-res tactile sensorsHigh res tactile sensorsExposed AuPassivated metal Exposed AuPassivated metal
Markers (METAL) for Piezo-polymer foilalignment
Markers (METAL) for Piezo-polymer foilalignment
1 cm
1 cm
Temperature Temperature psensors (Pwell Diode)
psensors (Pwell Diode)
Contact (transistor gates) + PVDF
FETs
1.0Normalised Taxels output
Contact (transistor gates) + PVDF
1 2 3 4 1
23
45
6
0.0
0.5
Y Axis
4 56
1X Axis
Testing the piezo- film deposition Cross-talk btw. taxels
By Ravinder Dahiya, Maurizio Valle (Unige) & Leandro Lorenzelli (Trento)
![Page 47: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/47.jpg)
With Paul Fitzpatrick and Lorenzo Natale
L l 1 P / b h
Cognitive Architecture
Level 2 APIs: Prospective Action Behaviors
Coordinated operation: Ontogenic Development
Level 1 APIs: perception/action behaviorsInnate perception/action primitives loose federation of behaviors
Based on phylogenic
configuration
own learning model
Software ArchitectureMultiple YARP processes
Running on multiple processors
Level 0 APIs: data acquisition & motor control
Running on multiple processors
Gbit Ethernet
DSP
iCubEmbeddedSystems
HUB
DSP DSP DSP
pc104
DSP SystemsDSP D
Sensors & Actuators
![Page 48: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/48.jpg)
more work in progress…more work in progress
• Ryo Saegusa: sensory predictionRyo Saegusa: sensory prediction• Andrew Dankers: models of vision
M tt F lli f t l• Matteo Fumagalli: force control• Boris Duran: dynamical systems control• Lorenzo Jamone: grasping• Serena Ivaldi: optimal controlSerena Ivaldi: optimal control• Massimiliano Izzo: internal models
![Page 49: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/49.jpg)
Lots of peopleLots of people• Lorenzo Natale, Francesco Nori: Software, testing, calibration• Marco Maggiali, Marco Randazzo: firmware, DSP libraries, tactile sensing• Francesco Becchi Paolo Pino Giulio Maggiolo Gabriele Careddu: design and Francesco Becchi, Paolo Pino, Giulio Maggiolo, Gabriele Careddu: design and
integration• Gabriele Tabbita, Walter Fancellu: assembly• Nikos Tsagarakis, William Hinojosa: legs and spine, force/torque sensors• Bruno Bonino, Fabrizio Larosa, Claudio Lorini: electronics and wiringun n n , F z L , u L n n n w ng• Luciano Pittera: wiring• Mattia Salvi: CAD maintenance• Alberto Zolezzi: managing quotes, orders and spare parts• Giovanni Stellin: hand• Ricardo Beira, Luis Vargas, Miguel Praca: design of the head and face• Paul Fitzpatrick & Alessandro Scalzo: software middleware• Alberto Parmiggiani: joint level sensing• Alexander Schmitz: fingertipsf g p• Nestor Nava: small Harmonic Drive integration• Ravinder Dahiya: FET-PVDF tactile senors• Lorenzo Jamone: fingertips• Daniel Roussy: constructiony• Ludovic Righetti: simulation and initial torque specification
![Page 50: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/50.jpg)
Other tunnelsOther tunnels
![Page 51: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/51.jpg)
Experimental setupExperimental setup…
![Page 52: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/52.jpg)
The initial idea…The initial idea…
![Page 53: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/53.jpg)
Objects come to existence jbecause they are manipulated
Fixate target Track visual motion…
(…including cast shadows)
Detect moment of impact
Separate arm, object motion
Segment object
Which edge should beWhich edge should be considered? Maybe some cruel
grad-studentglued the cube to the table
Color of cube and table are poorly separated
table
by Paul Fitzpatrick
Cube has misleading surface pattern
![Page 54: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/54.jpg)
Gesture “vocabulary”y
pull in
side tapside tap
back slappush away
![Page 55: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/55.jpg)
Exploring an affordance: rollingExploring an affordance: rolling
A toy car: it rolls in the direction of its principal axis
A bottle: it rolls orthogonal to the direction of its principal axis
A toy cube: it doesn’t roll, A ball: it rolls, it doesn’t have a principal axis it doesn’t have a principal axis
![Page 56: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/56.jpg)
Forming object clustersg j
![Page 57: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/57.jpg)
Into object affordances…j
0.4
0.5
0.4
0.5
0.2
0.3
0.2
0.3Bottle, “pointiness”=0.13 Car, “pointiness”=0.07
ccur
renc
e
Rolls at right Rolls
0 10 20 30 40 50 60 70 80 900
0.1
0 5
0 10 20 30 40 50 60 70 80 900
0.1
0 5babi
lity
of o
c angles toprincipal axis
along principal axis
0.3
0.4
0.5
0.3
0.4
0.5
Ball, “pointiness”=0.02Cube, “pointiness”=0.03
mat
ed p
rob
0
0.1
0.2
0
0.1
0.2estim
0 10 20 30 40 50 60 70 80 900 0 10 20 30 40 50 60 70 80 900
difference between angle of motion and principal axis of object [degrees]
![Page 58: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/58.jpg)
The geometry of pokingg y p g0.3
0.35
0.3
0.35
0.3
0.35
0.3
0.35
backslap pull in
0.1
0.15
0.2
0.25
0.1
0.15
0.2
0.25
0.1
0.15
0.2
0.25
0.1
0.15
0.2
0.25
ility
-200 -150 -100 -50 0 50 100 150 2000
0.05
-200 -150 -100 -50 0 50 100 150 2000
0.05
-200 -150 -100 -50 0 50 100 150 2000
0.05
-200 -150 -100 -50 0 50 100 150 2000
0.05
d pr
obab
i
0.15
0.2
0.25
0.15
0.2
0.25
0.15
0.2
0.25
0.15
0.2
0.25
side tap push away
Est
imat
ed
0.05
0.1
0.05
0.1
0.05
0.1
0.05
0.1
E
-200 -150 -100 -50 0 50 100 150 2000-200 -150 -100 -50 0 50 100 150 2000
direction of movement [deg]-200 -150 -100 -50 0 50 100 150 2000-200 -150 -100 -50 0 50 100 150 2000
![Page 59: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/59.jpg)
Behavior: poking according to affordanceto affordance
0.4
0.5
0.4
0.5Look for affordance
Searchrotation
0.1
0.2
0.3
0.1
0.2
0.3
affordance
identifyand
0 10 20 30 40 50 60 70 80 9000 10 20 30 40 50 60 70 80 900
localizeobject
P i l k dPreviously-pokedprototypes
Look forLook for action to satisfy
affordance
![Page 60: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/60.jpg)
Behavior: poking according to p g gaffordance
![Page 61: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/61.jpg)
Understanding a foreign manipulatorUnderstanding a foreign manipulator
Object, goal connects robot and human actionObject, goal connects robot and human action
![Page 62: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/62.jpg)
Interpreting observationsInterpreting observations“The robot could actually tell this was a side tap”
Initial position
Final position
A foreign manipulator (human) pokes an objectThe direction of movement is compared with the object affordance
![Page 63: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/63.jpg)
I t ti b tiInterpreting observationsInvoking the object’s natural Going against the object’s Invoking the object’s natural Going against the object’s
Demonstration by human
rolling affordance natural rolling affordance
Demonstration by human
rolling affordance natural rolling affordance
human
Mimicry in similar
human
Mimicry in similar situation
Mimicry when
situation
Mimicry whenMimicry when object is rotated Mimicry when object is rotated
![Page 64: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/64.jpg)
MimicryMimicry
![Page 65: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/65.jpg)
Data from human graspingData from human grasping2 cameras
FrameTo disk
ImagesFrame
grabbers
RS232
Cyber-glove
Other sensors
40 msec
Tracker
To disk
Tactile RS232
sensors
![Page 66: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/66.jpg)
Bayesian classifierBayesian classifier
{Gi}: set of gestures
168 sequences per subject10 subjects
6 complete sets{Gi}: set of gesturesF: observed features{Ok}: set of objects
z
6 complete sets
p(Gi|Ok): priors (affordances)
a
~ 76
cm
x
yz
p(F|Gi,Ok): likelihood to observe F
( ) ( ) ( ) ( )( ) ( ) ( ) ( )| , | , | / |i k i k i k kp G O p G O p G O p O=F F F
( )ˆ arg max | ,i
MAP i kG
G G O= F -45° (b)+90° (a) b
° ( )
+45° (b)+180° (a)
0° (b)+135° (a)
![Page 67: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/67.jpg)
f Two types of experiments
Vision Classifier
Fv, OkGi
Fv Ok Fm Ok
Vision ClassifierVMM
Fv, Ok Fm, OkGi
Learned by backpropagation ANN
![Page 68: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/68.jpg)
EstimationEstimation
p(Gi|Ok): affordances by counting estimated • p(Gi|Ok): affordances, by counting, estimated on the whole database
• p(F|Gi Ok): EM algorithm on the parameters p(F|Gi,Ok): EM algorithm on the parameters of a mixture of Gaussians (from Matlab implementation)
• VMM: Neural network, sigmoidal activation units, linear output, trained on the whole databasedatabase
![Page 69: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/69.jpg)
Role of motor information in action understanding
Object affordances (priors)
Visual space Motor space
Classification(recognition)Grasping actions
U d st di i s: bi b ti h G M tt G S di i L Understanding mirror neurons: a bio-robotic approach. G. Metta, G. Sandini, L. Natale, L. Craighero, L. Fadiga. Interaction Studies. Volume 7 Issue 2. 2006
![Page 70: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/70.jpg)
Some results…Exp. I(visual)
Exp. II(visual)
Exp. III(visual)
Exp. IV(motor)
Training# Sequences 16 24 64 24
# of view points 1 1 4 1
Cl ifi ti 100% 100% 97% 98%Classification rate
100% 100% 97% 98%
# Features 5 5 5 15# Modes 5-7 5-7 5-7 1-2
Test
# Sequences 8 96 32 96
# of view points 1 4 4 4p
Classification rate
100% 30% 80% 97%
![Page 71: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/71.jpg)
Speech againSpeech againSound
ead
Ultrasound
men
t of t
he h
eps
, ton
gue
Lary
nx
Mov
em li
Sound, video, etc.
Facial expression Shape of the tongue
With Michele Tavella & the CONTACT project team
![Page 72: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/72.jpg)
System-level approachSystem level approach
![Page 73: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/73.jpg)
Cognition
Cognition: a process by which a system achieves behaviour that is
Cognition
Cognition: a process by which a system achieves behaviour that is
- robust
- adaptive
- anticipatoryp y
- autonomous
Entails embodied perception, action, and interaction
![Page 74: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/74.jpg)
Our approachppGuiding Philosophy
– Cognition cannot be hand-coded – Is necessarily the product of a process of – Is necessarily the product of a process of
embodied development– Initially dealing with immediate events y g– Increasingly acquiring a predictive capability
Cognition and perception are functionally-dependent on the richness f th ti n int fof the action interface
![Page 75: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/75.jpg)
Our Approach (contd )Our Approach (contd.)
Emergent embodied cognitive systems:Emergent embodied cognitive systems:
Gi i h t f i t ti d ti • Given a rich set of innate action and perception capabilities
• Develop over time an increasing range of cognitive Develop over time an increasing range of cognitive abilities
• Recruiting ever more complex actions• Achieving an increasing degree of prospection
(and, hence, adaptability and robustness)
![Page 76: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/76.jpg)
RobotCub Cognitive ArchitectureRobotCub Cognitive Architecture
Grounded in neuroscience and psychology
R t d i ti d d t ti Rooted in action-dependent perception
Focused on adaptive & prospective capabilities
Designed to facilitate development
Cognitive architecture ≡ (RobotCub) PhylogenyCognitive architecture (RobotCub) Phylogeny
![Page 77: From natural to artificial cognitive systems](https://reader035.fdocuments.us/reader035/viewer/2022071602/613d65d2736caf36b75cdaa4/html5/thumbnails/77.jpg)
In spite of the wiring problems…