iCub: the open‐source humanoid robot…

Giorgio MettaUniversity of Genoa and Italian Institute of Technology

The tale of the Wright brothersThe tale of the Wright brothers(and three messages)

1 Reverse engineering:1. Reverse engineering:– Looking and copying bird flight → aircraft design– Reverse of reverse engineering → aerodynamics led to

better understanding of bird flight (forward engineering)2. Models:

– Wright started from a previous model of Lilienthal (which was wrong) → but then they had (after 2 years) to produce their own models (and test them) they built a wind tunnel their own models (and test them), they built a wind tunnel (very modern)!

3. Stability and control:• Separate models didn’t work well (either stability or Separate models didn t work well (either stability or

control)• Discovered that the key to stability and control is by

rolling → turn by rolling! Separate models don’t work, system wide approach is requiredsystem-wide approach is required.

• Understanding at the systems level

and the story goes…and the story goes

• Late 1903 first powered flight (35m • Late 1903, first powered flight (35m, 10km/hour)5 l t 2 h fli ht• 5 yrs later, 2 hours flight

• 8 yrs later, across North America• 24 yrs later, New York to Paris• 65 yrs later three people to the moon65 yrs later, three people to the moon• Now, small seats and screaming

infantsinfants

Three messagesThree messages

• Reverse engineering:Reverse engineering:– Study and be inspired by the brainModels and experiments “wind tunnels”:• Models and experiments wind tunnels :– Mathematical models and robotsGl b l h• Global approach:– Sanity check by implementing everything

on a real physical platform, complete systems, real feedback

Infants in 10‐17m range are very challanged when you put a toy in a drawer and 

l it If th th illclose it. If they can, they will immediately open the 

drawer and take out the toy. Our drawer resisted openingOur drawer resisted opening 

by a weight attached to it.

ll h h ll h f h bPulling something that resists pulling has to start from the baseof support. Adults will start the pull by activating the gastrocnemious muscles 50 ms before the arm starts pulling.

von Hofsten C., Rosander K., et al.

10-month-old infant10-month-old infant 16-month-old infant16-month-old infant

Drawer starts movingDrawer starts moving Drawer starts movingDrawer starts moving

From: Lackner JR Dizio P Gravitoinertial force background level affects adaptation toFrom: Lackner JR, Dizio P. Gravitoinertial force background level affects adaptation to Coriolis force perturbations of reaching movements. Journal of Neurophysiology 1998, 80:546‐553.

Forwardmodel

Efferentcopy

Gripforce

controller

RealizedDesired

trajectory Inverse

Fingers Gripforce

controller

Arm Realizedtrajectory

j y Inversemodel

From: Flanagan JR, Wing AM. The role of internal models in motion planning andFrom: Flanagan JR, Wing AM. The role of internal models in motion planning and control: evidence from grip force adjustments during movements of hand‐held loads.Journal of Neuroscience 1997, 17:1519‐1528.

Experimental setup

Stimuli (3 conditions)

With Alessandra Sciutti, Francesco Nori, Thierry Pozzo

ResultsResults

Results (2): Slopes0.070.07

Results (2): Slopes

0.05

0.06

0.05

0.06

p = 0 217

0.03

0.04

Slo

pes

0.03

0.04

Slo

pes

p   0.217

No significant difference among slopes in all 

0.01

0.02

0.01

0.02g p

conditions.

Fixed down Fixed up Variable Variable only down Variable only up0

ConditionsFixed down Fixed up Variable Variable only down Variable only up

0

Conditions

Is it kinematics or dynamics h ?that matter?

Scheidt, Robert A., David J. Reinkensmeyer, Michael A. Conditt, W. Zev Rymer, and Ferdinando A. Mussa-Ivaldi Persistence of motor adaptation during constrained multi-joint arm movements JMussa-Ivaldi. Persistence of motor adaptation during constrained, multi-joint, arm movements. J Neurophysiol 84: 853–862, 2000.

Looking at othersLooking at others

H-reflex deviation from the mean(conductance of the spinal-muscle nerves)

Muscle anticipate the kinematics

kinematics

Borroni P Montagna M Cerri G & Baldissera F (2005) Cyclic time course

Prone position, wrist ext/flexion (ECR, FCR muscles)

Borroni, P., Montagna, M., Cerri, G., & Baldissera, F. (2005). Cyclic time course of motor excitability modulation during the observation of a cyclic hand movement. Brain Research, 1065, 115-124.

Supine positionSupine position

As before, but differentphase difference btw thephase difference btw thekinematics and the muscularactivation

Then compare with actual actionThen compare with actual action

Which was 54° on average EMG signals fromthe FCR and ECRmuscles

Which was 112° on average

The movement doesn’t change

Coherence of motion…

@LIRA-Lab 1991

Objects come to existence jbecause they are manipulated

Fixate target Track visual motion…

(…including cast shadows)

Detect moment of impact

Separate arm, object motion

Segment object

Which edge should beWhich edge should be considered? Maybe some cruel

grad-studentglued the cube to the table

Color of cube and table are poorly separated

table

With Paul Fitzpatrick

Cube has misleading surface pattern

Into object affordances…j

0.4

0.5

0.4

0.5

0.2

0.3

0.2

0.3Bottle, “pointiness”=0.13 Car, “pointiness”=0.07

ccur

renc

e

Rolls at right Rolls

0 10 20 30 40 50 60 70 80 900

0.1

0 5

0 10 20 30 40 50 60 70 80 900

0.1

0 5babi

lity

of o

c angles toprincipal axis

along principal axis

0.3

0.4

0.5

0.3

0.4

0.5

Ball, “pointiness”=0.02Cube, “pointiness”=0.03

mat

ed p

rob

0

0.1

0.2

0

0.1

0.2estim

0 10 20 30 40 50 60 70 80 900 0 10 20 30 40 50 60 70 80 900

difference between angle of motion and principal axis of object [degrees]With Paul Fitzpatrick

The qualitative “geometry” of pokingq g y p g

0.350.35

backslap pull in

0 1

0.15

0.2

0.25

0.3

0 1

0.15

0.2

0.25

0.3 backslap

abili

ty

pull in

-200 -150 -100 -50 0 50 100 150 2000

0.05

0.1

-200 -150 -100 -50 0 50 100 150 2000

0.05

0.1

0 250 25ated

pro

b

side tap

0 1

0.15

0.2

0.25

0 1

0.15

0.2

0.25

side tap

Est

ima

-200 -150 -100 -50 0 50 100 150 2000

0.05

0.1

-200 -150 -100 -50 0 50 100 150 2000

0.05

0.1

back slappush awayDirection of movement [deg] back slappush away

Behavior: using affordancesBehavior: using affordances

I t ti b tiInterpreting observationsInvoking the object’s natural Going against the object’s Invoking the object’s natural Going against the object’s

Demonstration by human

rolling affordance natural rolling affordance

Demonstration by human

rolling affordance natural rolling affordance

human

Mimicry in similar

human

Mimicry in similar situation

Mimicry when

situation

Mimicry whenMimicry when object is rotated Mimicry when object is rotated

Behavior: mimicrym m y

Data from human graspingData from human grasping2 cameras

FrameTo disk

ImagesFrame

grabbers

RS232

Cyber-glove

Other sensors

40 msec

Tracker

To disk

Tactile RS232

sensors

Bayesian classifierBayesian classifier{Gi}: set of gestures

168 sequences per subject10 subjects

6 complete setsF: observed features{Ok}: set of objects

z

6 complete sets

p(Gi|Ok): priors (affordances)p(F|Gi Ok): likelihood to observe

a

~ 76

cm

x

yz

p(F|Gi,Ok): likelihood to observe F

( ) ( ) ( ) ( )( ) ( ) ( ) ( )| , | , | / |i k i k i k kp G O p G O p G O p O=F F F

( )ˆ arg max | ,i

MAP i kG

G G O= F -45° (b)+90° (a) b

° ( )

+45° (b)+180° (a)

0° (b)+135° (a)

f Two types of experiments

Vision Classifier

Fv, OkGi

Fv Ok Fm Ok

Vision ClassifierVMM

Fv, Ok Fm, OkGi

Learned by backpropagation ANN

EstimationEstimation

p(Gi|Ok): affordances by counting estimated • p(Gi|Ok): affordances, by counting, estimated on the whole database

• p(F|Gi Ok): EM algorithm on the parameters p(F|Gi,Ok): EM algorithm on the parameters of a mixture of Gaussians (from Matlab implementation)

• VMM: Neural network, sigmoidal activation units, linear output, trained on the whole databasedatabase

Role of motor information in action understanding

Object affordances (priors)

Visual space Motor space

Classification(recognition)Grasping actions

U d st di i s: bi b ti h G M tt G S di i L Understanding mirror neurons: a bio-robotic approach. G. Metta, G. Sandini, L. Natale, L. Craighero, L. Fadiga. Interaction Studies. Volume 7 Issue 2. 2006

Some results…Exp. I(visual)

Exp. II(visual)

Exp. III(visual)

Exp. IV(motor)

Training# Sequences 16 24 64 24

# of view points 1 1 4 1

Cl ifi ti 100% 100% 97% 98%Classification rate

100% 100% 97% 98%

# Features 5 5 5 15# Modes 5-7 5-7 5-7 1-2

Test

# Sequences 8 96 32 96

# of view points 1 4 4 4p

Classification rate

100% 30% 80% 97%

Learning how to graspLearning how to grasp

Question: How well do we guess the subject’s grasping style? show the system the initial part of the hand trajectory (22 joints + 6 position/orientation) and figure out the final

fi ticonfiguration

Learning: SVM-regression With C. Castellini, F. Orabona

Learning how to graspLearning how to grasp• Allows for a precise prediction ofAllows for a precise prediction of

– hand position: 1.5cm– hand orientation: 2 5°hand orientation: 2.5– hand posture: 7.5°

• Reasonably predictive:• Reasonably predictive:– hand position: 200ms

h d i t ti 120– hand orientation: 120ms– hand posture: 90-200ms (depending on

th bj t)the object)

Learning how to graspLearning how to grasp

Affordance: knowing the object will uniformly improve the error, at no additional computational overhead (it requires clustering of the data which can be supervised by object recognition)supervised by object recognition)

Future developmentFuture development…Speech

he h

ead

ueem

ent o

f th

lips,

tong

u

Lary

nx

Mov

e

Facial expression Shape of the tongue

The mirror system and speech: with L. Fadiga, L. Craighero

Speech listeningSpeech listening…

The iCub: quick summaryq yThe iCub is the humanoid baby-robot

designed as part of the RobotCub project

– The iCub is a full humanoid robot sized as a three and half year-old child.f y

– The total height is 104cm.– It has 54 degrees of freedom, including

articulated hands to be used for manipulation articulated hands to be used for manipulation and gesturing.

– The robot will be able to crawl and sit and autonomously transition from crawling to autonomously trans t on from crawl ng to sitting and vice-versa.

– The robot is GPL/FDL: software, hardware, drawings, documentation, etc.g , ,

Degrees of freedomDegrees of freedom

• Head: vergence, common tilt + 3 dof neck• Arms: 7 dof each

Sh ld (3) lb (1) i t (3)– Shoulder (3), elbow (1), wrist (3)• Hands: 9 dof each ► 19 joints

– 5 fingers ► underactuated• Legs: 6 dof each

– Hip (3), knee (1), ankle (2)• Waist: 3 dofWaist: 3 dof

Σ = 53 dof (not counting the facial expressions)

SensorizationSensorization• Cameras

– Pointgrey Dragonfly firewires

80x30mm

cameras• Force/torque sensors

– Custom development: 6 axialMi h k

58x42mm

• Microphones, speaker– Standard condenser electrect

miniature microphones– Pinnae– Pinnae

• Gyroscopes, linear accelerometers– Xsense: Mtx

• Joint level sensing• Joint level sensing– Position sensors, current consumption,

temperature• Custom electronicsCustom electronics

– DSP based, programmable embeddedcontrollers

Facial expressionsFacial expressions

Body cover: conceptBody cover: concept

The iCubThe iCub

Preprogrammed movements Preprogrammed movements with collision limits

More examples…

With Peter Ford-Dominey (U. Lyon 2) With Auke Ijspeert, Ludovic Righetti, Sarah Degallier (EPFL)

With a lot of students @ RobotCub summer school 2008 With VisLab (IST Lisbon)

WikiCVS

Part lists

Drawings

Promoting the iCubPromoting the iCub• RobotCub Open Call

31 ti i t 6 i ill i f th – 31 participants, 6 winners will receive a copy of the iCub free of charge

• Further developmentFurther development– FP7 project ITALK: 4 iCub’s will be built, language

• CollaborationsCollaborations– Univ. of Karlsruhe: new and longer legs

• Simulator:Simulator– Both Open Source and as a model in Webots

Robot OneRobot One

Robot TwoRobot Two

Robot ThreeRobot Three

Robot FourRobot Four

Robot Five and SixRobot Five and Six

Next: the skinNext: the skinPrinciple

Lot of sensing pointsPatented

Structure of the skin

FingertipFingertip

OutlineOutline

Silicon cover Electronics

3D CAD Electrode fabrication Complete prototype3D CAD Electrode fabrication

PeoplePeople• Giulio Sandini: Mentor & guidance• Lorenzo Natale, Francesco Nori: Software, testing, calibration• Marco Maggiali Marco Randazzo: firmware DSP libraries tactile sensingMarco Maggiali, Marco Randazzo: firmware, DSP libraries, tactile sensing• Francesco Becchi, Paolo Pino, Giulio Maggiolo, Gabriele Careddu: design and

integration• Roberto Puddu, Gabriele Tabbita, Walter Fancellu: assembly• Nikos Tsagarakis, William Hinojosa: legs and spine, force/torque sensorsN g , W m H n j g n p n , f / qu n• Bruno Bonino, Fabrizio Larosa, Claudio Lorini: electronics and wiring• Luciano Pittera, Davide Dellepiane: wiring• Mattia Salvi: CAD maintenance• Alberto Zolezzi: managing quotes, orders and spare partsm g g q , p p• Giovanni Stellin: hand• Ricardo Beira, Luis Vargas, Miguel Praca: design of the head and face• Paul Fitzpatrick & Alessandro Scalzo: software middleware• Alberto Parmiggiani: joint level sensinggg j g• Alexander Schmitz: fingertips• Nestor Nava: small Harmonic Drive integration• Ravinder Dahiya: FET-PVDF tactile senors• Lorenzo Jamone: fingertipsg p• Jean-Baptiste Keller, Daniel Roussy: construction• Ludovic Righetti: simulation and initial torque specification

In spite of the wiring problems…