Projection Computer Graphics

on Moving Surfaces: A Simple Calibration and Tracking Method

Claudio Pinhanez MIT Media Laboratory

Frank Nielsen, Kim Binsted Sony Computer Science Laboratory

Motivations

transforming every surface in

a space into a display

movable displays

enhancing performance (theater, story-telling, dance)

Structure of the Problem

range

Goal: Correction of the Projection

when the surface moves, the projection is corrected

Problems

1. Calibrate the projector/camera relation

2. Track position/attitude of the surface

3. Correct in real time the projection

4. Compensate for oblique projection

[Raskar, Sketches’99]

Problem 2.a:

Tracking the Position of the Surface

hard for vision-based

systems

(darkness)

“hacked” solution:

4 Infra-red LEDs

5th disambiguating LED

5th marker

Problem 1: Calibration

From stereo vision:

the relationship between points on a planar

surface seen from 2 different cameras is a

homography (3x3 linear matrix) [Faugeras 93]

Geometrically,

pinhole cameras = pinhole projectors

Camera/Projector Homography

camera projector

homography

c

c

c

p

p

p

z

y

x

HHc

z

y

x

p

cccc zyzxc , pppp zyzxp ,

Calibration Process

Only 4 points!

Given:

Read:

Since P=HC,

4,3,2,11,, iyxp i

p

i

pi

4,3,2,11,, iyxc i

c

i

ci

1 TT CCPCH

TTTT ppppP 4321 TTTT ccccC 4321

(manual alignment)

Problem 3:

Correcting the Projection

p=Hc

Delay and Noise

delay:

position at time t

projected at time t+dt (100-200ms)

sensor noise

Solution: Kalman filter

Problems

1. Calibrate the projector/camera relation

2. Track position/attitude of the surface

3. Correct in real time the projection

4. Compensate for oblique projection

[Raskar, Sketches’99]

Problem 2.b: Estimating the Attitude

4 co-planar points -> unique solution

(4-perspective inversion)

high degree polynomial, very sensitive to physical

parameters -> unstable solution

attitude computed from ratio of

line segments + linear regression

in practice, better and faster

than iterative solution 1/1=1 0.7/1.4=0.5

HyperMask [Millenium Motel]

white facial mask

with IR markers camera

projector

mask

Attitude/Position Tracking

Multiple Characters

Facial Expressions

Siggraph’ 99 Video

(Lip Synching)

sound-based predictive lip synching

neural network: predicts the position of

mouth from the sound stream

Shigeo Morishima (Seikei University)

HyperMask: Performances

Tomorrow, 9 am to 1 pm

Emerging Technologies

Millenium Motel

Questions and Answers(?)

More info:

http://www.csl.sony.co.jp/

person/nielsen/HYPERMASK/hypermask.html

?

?

? ?

? ?