Vision-Based Reach-To-Grasp Movements

From the Human Example to an Autonomous Robotic System

Alexa Hauck

Context

Special Research Program “Sensorimotor”

C1: Human and Robotic Hand-Eye Coordination

• Neurological Clinic (Großhadern), LMU München

• Institute for Real-Time Computer Systems, TU München

MODEL

ofHand-Eye

CoordinationAN

ALY

SIS

of

hum

an r

each

ing m

ovem

ents

SYN

TH

ESIS

of

a r

oboti

c sy

stem

The Question is ...

How to use which visual information for motion control?

control strategy representation catching reaching

State-of-the-art Robotics

)(),(),( txtxtxxT

+ easy integration with path planning

+ only little visual information needed– sensitive against model errors

)())(( txtxxT

+ model errors can be compensated

– convergence not assured

– high-rate vision needed)())(( txtffT

Impressive results

... but nowhere near human performance!

Visual Servoing: (visual feedback control)

Look-then-move: (visual feedforward control)

The Human Example

Separately controlled hand transport:• almost straight path• bell-shaped velocity profile

Experiments with target jump:• smooth on-line correction of the trajectory

Experiments with prism glasses:• on-line correction using visual feedback • off-line recalibration of internal models

Use of visual information in spatial representation Combination of visual feedforward and feedback

... but how ?

New Control Strategy

1

1

))()(()())(()()(n

iniiiTnnn tgtgteDtxxtgttx

Example: Point-to-point

Example: Target Jump

Example: Target Jump

Example: Target Jump

Example: Multiple Jumps

Example: Multiple Jumps

Example: Double Jump

Hand-Eye System

Robotimages

ImageProcessing

features

ImageInterpretation

position target & hand

MotionPlanning

trajectory

RobotControl

commands

Models

Hand-EyeSystem

&Objects

objectmodel

sensormodel

armmodel

objectmodel

The Robot: MinERVA

manipulator with 6 joints

CCD cameras

pan-tilt head

Robot Vision

3D

Bin. Stereo

Target

correspondingpoints

Hand

correspondingpoints

Example: Reaching

Example: Reaching

Example: Reaching

Model Parameters

Arm:• geometry, kinematics• 3 parameters

Arm-Head Relation:• coordinate transformation• 3 parameters

Head-Camera Relations:• coordinate transformations• 4 parameters

Cameras:• pinhole camera model• 4 parameters (+ rad. distortion)

Calibration

manufacturer

measuring tape

HALCON

HALCON

Use of Visual Feedback

mean maxcorr0 8.9cm 20cm

1 Hz 0.4cm 1cm

Example: Vergence Error

Example: Compensation

Summary

• New control strategy for hand-eye coordination

• Extension of a biological model

• Unification of look-then-move & visual servoing

• Flexible, economic use of visual information

• Validation in simulation

• Implementation on a real hand-eye system