Hand-Eye Coordination and Vision-based Interaction http://www.cs.ualberta.ca/~jag/courses/HandEye/

cmput610

Martin Jagersand

Today:

1. Fun (hopefully) intro to some of the topics in the course. Give flavor of course.

2. Suggestions for some exciting projects you can choose. You are also welcome to propose your own.

3. (Boring) administrative details: Course schedule, room, participation and examination, class list etc…

Main focus of course

Most other courses are about static vision. What is in the image?

This course: Vision for regestering, understanding and generating motion. How to act based on visual information.

Im Processing Symbol Interp

Images

Agent doing physical motion

Real world

Where/What or Action/Perception division also in biological vision

Main topics we will cover

1. Motion in the real world and its image projection

2. Vision and other sensory based control

3. Biological motor control.

4. User interfaces for computers, robots and other machines.

Motion estimation:Image subtraction

50

Candidate areas formotion

Detecting motion:

Motion estimation 2

A vector flow is a more useful representation

Motion estimation 3

Non-rigid motions are also important

Motion estimation 4

Generalize to many freedooms (DOFs)

One application: Tracking

Goal: Stabilizing motion. Find move params

Vision and Sensory based motion control

Uncalibrated Visual Servoing

Let y = visual observation; x = motor control Linear system model:

y= Linear p-controller

Estimate the Visual-Motor Jacobian

Jêk+1 = Jêk+ É xTÉ x

(É ymeasuredà JêkÉ x)É xT

f (x) ù f (xk) + J (xk)(x à xk)

xç = K J +kyç

Shortcut in Image based visual servoing

Biological sensory-motor control

Sensory and motor areas in cortex

Example: Eye movements

Relatively well studied area Yet complex nonlinear kinematics Interesting adaption behaviour

Paths: 1 flash Multple targets Remembered location

User Interface applications:Gesture and motion tracking

User Interfaces 2:Robot Control

Vision based “Tele Assistance”

Describe task and objects by gestures and pointing

Visual language maps to physical actions

Composite Task: Solving a Puzzle

Proposed projects 1

Neurosci Georgopoulos, Kalaska, Shieber...

PercAct Goodale, Lomis,

Characterizing motion: Activity recognition: Nelson,Alimonios Fleet, Jepson... Jagersand Matlab implementaton

Animating motion. Gleicher, Jagersand, Thalman

Proposed projects 2

Tracking: Hager, Black, XVision: practical experiments with. Filtering, Bar-Shalom:

Robotics: Controller HW and SW for the small robot arm (or the PUMA's)

Real time systems Real time linux

Participation and Examination

Course readings Class discussion, active class participation Present some papers of your choice Do your own reading or practical project and a

writeup

Administrative

Register for cmput610 Proposed schedule: Tue, Thu 9-10am Room TBA Course literature and readings are given on-line when

possible. Announcements via email. Make sure you are on the

email list. Course web page (important):

http://www.cs.ualberta.ca/~jag/courses/HandEye/