Projector with Radiometric Screen

Compensation

Shree K. Nayar, Harish Peri

Michael Grossberg, Peter Belhumeur

Support:National Science Foundation

Computer ScienceColumbia University

Procams WorkshopICCV 2003, Nice, France

Projecting on Any Surface ?

Textured Screen

Camera

Computer

Projector

Projector-Camera System

Geometric Calibration

Projector Input Camera Output

Maximum Error: 0.6 pixels, RMS Error: 0.18 pixels

Display Device

Projector

ScreenI D

P

E

xixd

xs

CameraCapture Device

BM

CL

xbxm

Projector-Camera Pipeline: Radiometry

What Display Image (I) will produce a Desired Captured Image (M) ?

Radiometric Model (for each pixel)

Display Device

Projector

ScreenI DP

E

CameraCapture Device

BMC L

))(( KKKK IdpP

display reponseprojector reponse

))(( 11LLLL MmbC

capture reponsecamera reponse

dqswV LKKL )()()(projector channel

screen camera channel

B

G

R

BBBGBR

GBGGGR

RBRGRR

B

G

R

P

P

P

VVV

VVV

VVV

C

C

C

camera irradiance projector brightness

color mixing matrix

Finding the Model Parameters (at each pixel)

Display Device

Projector

ScreenI DP

E

CameraCapture Device

BMC L

B

G

R

BBBGBR

GBGGGR

RBRGRR

B

G

R

P

P

P

VVV

VVV

VVV

C

C

C

camera irradiance projector brightness

color mixing matrix

)(

)(

)(

1

1

1

BBB

GGG

RRR

PgI

PgI

PgI

display image color projector color

composite response

B

G

R

BGBR

GBGR

RBRG

B

G

R

P

P

P

VV

VV

VV

C

C

C

1

1

1

camera irradiance projector brightness

modified color mixing matrix

(known)

Finding the Color Mixing Matrix

B

G

R

BGBR

GBGR

RBRG

B

G

R

P

P

P

VV

VV

VV

C

C

C

1

1

1

camera irradiance projector brightness

modified color mixing matrix

(known)

arbitrary image only change in red

1I 2I

RR CP

R

GGR C

CV

R

BBR C

CV

only change in green only change in blue

3I 4I

GG CP

G

RRG C

CV

G

BBG C

CV

BB CP

B

RRB C

CV

B

GGB C

CV

00.10134.0012.0

0363.000.10361.0

0010.05534.000.1

00.10253.00172.0

0652.000.10344.0

0010.00434.000.1

00.10218.00094.0

0500.000.10471.0

0086.05973.000.1

00.10588.00119.0

0465.000.10357.0

0001.06764.000.1

00.10213.00100.0

0476.000.10500.0

0095.05561.000.1

00.10195.00338.0

0632.000.10593.0

0012.05804.000.1

00.10278.00217.0

0425.000.10543.0

0010.06972.000.1

00.10321.00367.0

1016.000.10461.0

0169.04545.000.1

Color Mixing: V Matrices

Radiometric Calibration: Color

Projector Input Camera Output

)(

)(

)(

1

1

1

BBB

GGG

RRR

PgI

PgI

PgI

desired projector output

compensated display image

composite response

Texture of Screen

Color Experiment

Uncompensation outputs (flat gray inputs: 100, 150, 200 gray levels)

Compensation images (flat gray inputs: 100, 150, 200 gray levels)

Compensated outputs (flat gray inputs: 100, 150, 200 gray levels)

Compensation Accuracy

50,50,50

100,100,100

150,150,150

200,200,200

25,26,26

66,70,71

75,98,95

50,99,90

2,3,7

6,8,14

19,15,21

39,29,25

24.1,24.2,24.5

58.9,57.2,64.9

49.0,44.5,69.1

27.7,47.5,42.7

0.6, 0.8, 1.7

1.6, 1.9, 5.4

4.8, 3.1, 5.8

12.7,4.0,10.7

ProjectedBrightness

Max. Error RMS Error

Without Comp.

With Comp. (R, G, B)

(R, G, B) (R, G, B)

Without Comp.

With Comp.

Original (desired) image (I) Uncompensated output (M)

Face Image on Checkerboard Screen

Compensated output (M)~

Compensation image ( I )~

Face Video on Brick Texture

Normal Projector Projector With Compensation

Scene Video on Brick Texture

Normal Projector Projector With Compensation

Face Video on Butterfly Poster

Normal Projector Projector With Compensation

Summary

• Off-Line Method for Radiometric Calibration

• On-Line Algorithm for Screen Compensation

• Refinement using Continuous Feedback

• Radiometric Model for Projector-Camera System