Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

42
Uncalibrated Image-based Uncalibrated Image-based Control of Robots Control of Robots Azad Shademan PhD Candidate Computing Science, University of Alberta Edmonton, Alberta, CANADA [email protected]

description

Visiting Lecturer Program (140) Speaker: Azad Shademan Ph.D. candidate Department of Computing Sciences University of Alberta, Canada Title: Uncalibrated Image-Based Robotic Visual Servoing Local Host: Ms. Nasim Pouraryan Time: Wednesday, November 5, 2008, 12:30-2:00 pm Location: Faculty of Electrical and Computer Engineering, University of Tehran, Tehran Abstract: Design of versatile vision-based robotic systems demands a solution with little or no dependence on system parameters. The problem of real-time vision-based control of robots has been long studied as robotic visual servoing. Most provably stable solutions to this problem require calibrated kinematic and camera models, because in a precisely calibrated system one can model the visual-motor function analytically. The uncalibrated approach has received limited attention mainly because the stability analysis is not as straightforward as that of calibrated image-based architecture. In an uncalibrated system the visual-motor function is not known, but partial derivative information (Jacobian) can be learned by tracking visual measurements during motion. In this talk, we study the uncalibrated image-based visual servoing and present different Jacobian learning methods.

Transcript of Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Page 1: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Uncalibrated Image-based Uncalibrated Image-based Control of RobotsControl of Robots

Azad ShademanPhD Candidate

Computing Science, University of AlbertaEdmonton, Alberta, CANADA

[email protected]

Page 2: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Vision-Based Control

current

desired

Left Image Right Image

A

A

A

B

BB

Page 3: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Vision-Based Control

Left Image Right Image

B

BB

Page 4: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Where is the camera located? Eye-to-Hand

e.g.,hand/eye coordination

Eye-in-Hand

Page 5: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Vision-Based Control Feedback from visual sensor (camera) to

control a robot Also called visual servoing Visual servoing is the task of minimizing a

visually specified objective by giving appropriate control commands to a robot

Is it any difficult?Images are 2D, the robot workspace is 3D 2D data 3D geometry

Page 6: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Visual Servo Control law Position-Based:

Robust and real-time pose estimation + robot’s world-space (Cartesian) controller

Image-Based:Desired image features seen from cameraControl law entirely based on image features

Hybrid:Depth information is added to image data to

increase stability

Page 7: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Position-Based Robust and real-time relative pose

estimation Extended Kalman Filter to solve the

nonlinear relative pose equations. Cons:

EKF is not the optimal estimator.Performance and the convergence of pose

estimates are highly sensitive to EKF parameters.

Page 8: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Position-Based

Desired pose

Estimated pose

Page 9: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Position-Based

Measurement noise

State variable

Process noise

yaw pitch roll

Measurement equation (projection) is nonlinear and must be linearized.

Page 10: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

K

xk-1,k-1

zk

Rk

Pk,k

Pk,k-1

Ck

Qk-1

xk,k

xk,k-1

Pk-1,k-1

Kalman Gain

Measurement noise covariance

A priori error cov. @ k-1

Process noise covariance

Initial/previous state

Linearization

Measurement

State update

State prediction Error cov. prediction

Error cov. update

Page 11: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Image-Based

Desired Image feature

Extracted image feature

Page 12: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Visual-motor Equation

x1

x2

x3

x4

q=[q1 … q6]

Visual-Motor Equation

This Jacobian is important for motion control.

Page 13: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Visual-motor JacobianImage spacevelocity

Joint spacevelocity

A

A

B

B

Page 14: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Image-Based Control Law

1. Measure the error in image space

2. Calculate/Estimate the inverse Jacobian

3. Update new joint values

Page 15: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Image-Based Control Law

Desired Image feature

Extracted image feature

Page 16: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Jacobian calculation Analytic form available if model is known.

Known model Calibrated

Must be estimated if model is not known

Unknown model Uncalibrated

Page 17: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Calibrated: Interaction Matrix

Analytic form depends on depth estimates.

Camera/Robot transform required. No flexibility.

CameraVelocity

Page 18: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Uncalibrated: Visual-Motor Jacobian

A naïve method: Orthogonal projections

Page 19: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Uncalibrated: Visual-Motor Jacobian

A naïve method: Orthogonal projections

Page 20: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Uncalibrated: Visual-Motor Jacobian

A naïve method: Orthogonal projections

Page 21: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Uncalibrated: Visual-Motor Jacobian

A popular local estimator:

Recursive secant method (Broyden update):

Page 22: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Relaxed model assumptions

Traditionally: Local methods No global planning (-) Difficult to show

asymptotic stability condition is ensured (-)

Main problem of traditional methods is the locality.

Calibrated vs. Uncalibrated

Model derived analytically Global asymptotic

stability (+) Optimal planning is

possible (+) A lot of prior knowledge

on the model (-)

Global Model Estimation (Research result)

Optimal trajectory planning (+)

Global stability guarantee (+)

Page 23: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Synopsis of Global Visual Servoing Model Estimation (Uncalibrated) Visual-Motor Kinematics Model Global Model

Extending Linear Estimation (Visual-Motor Jacobian) to Nonlinear Estimation

Our contributions:K-NN Regression-Based EstimationLocally Least Squares Estimation

Page 24: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Local vs. Global

Key idea: using only the previous estimation to estimate the Jacobian

1. RLS with forgetting factor Hosoda and Asada ’94

2. 1st Rank Broyden update: Jägersand et al. ’97

3. Exploratory motion: Sutanto et al. ‘98

4. Quasi-Newton Jacobian estimation of moving object: Piepmeier et al. ‘04

Key idea: using all of the interaction history to estimate the Jacobian

Globally-Stable controller design

Optimal path planning Local methods don’t!

Page 25: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

3 NN

K-NN Regression-based Method

q1

q2

x1

q1

q2

?

Page 26: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

q1

q2

x1 ?

K-neighbour(q)

(X,q)

Locally Least Squares Method

Page 27: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Eye-to-hand Experiments

Puma 560 Stereo vision Features: projection of the end-effector’s

position on image planes (4-dim) 3 DOF for control

Page 28: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Measuring the Estimation Error

Page 29: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Global Estimation Error

Page 30: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Visual Task Specification Image Features:

Geometric primitives (points, lines, etc.) Higher order image moments Shape parameters …

Visual Tasks Point-to-point (point alignment) Point-to-line (colinearity) Point-to-plane (coplanarity) …

Page 31: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Eye-in-hand

Page 32: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Eye-in-Hand Experiments

Page 33: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Eye-in-Hand Experiments

Page 34: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Eye-in-Hand Experiments

Page 35: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Eye-in-Hand Experiments

Page 36: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Mean-Squared-Error

Task 1

Task 2

Page 37: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Task Errors

Page 38: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Page 39: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Conclusions Reviewed position-based and image-based visual

servoing schemes. Presented two global methods to learn the visual-

motor function. KNN suffers from the bias in local estimations. LLS (global) works better than the KNN (global) and

local updates. The Jacobian of more complex visual tasks can also

be learned using LLS method.

[email protected]

Page 40: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Thank you!

Page 41: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Visual Ambiguity: Single Camera

Page 42: Uncalibrated Image-Based Robotic Visual Servoing (knowdiff.net)

Azad Shademan, Uncalibrated image-based control of robots Nov. 5, 2008

Visual Ambiguity: Stereo