Generalized Mosaics

Yoav Y. Schechner,

Shree Nayar

Department of Computer Science

Columbia University

Mosaics

Processing

Redundant Measurements

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Scanning with Less Redundancy

Mantis Shrimp

TakataC

Different rows = Different optical characteristics

Generalized Mosaicing

Field of View

BrightnessDynamic Range Spectrum

Depth of Field

Polarization

camera

Spatially varying

filter

Schechner, Nayar, Generalized mosaics

Mosaic + High Dynamic Range

88 - 18,794

Fusion of Measurements

)(Mask),(),(Raw xyxIyxk

1

22 1

k kI

I

k k

k

I

III

22

Maximum-Likelihood Solution

),( ),( )(Mask

),(RawyxIyxI

x

yxkk

k

Schechner, Nayar, Generalized mosaics

8 bit camera Dynamic range as 16 bits

Generalized Mosaicing: Yoav Schechner and Shree Nayar

8 bits (almost)(pixels)

-2

-4

-6

-8 x

log M2

Log of the mask

==1 10-61 10-2

M

x

average row

frames y

frameyxx ),(Raw )(Mask

),(Mask),(),(Raw yxyxIyxframe

Mask Self-CalibrationUnknown filter (vignetting)

consistencyconstraints

0Raw)(Mask Raw)(Mask1221 xx

MaskttMaskmin AA

Consistency, Smoothness A

Schechner, Nayar, Generalized mosaics

Image Registration

2pair ingcorrespondeach

pixels all

2total DIFFDIFF

2total DIFF:Minimize

2pair ingcorrespondeach

pixels all

2total DIFFDIFF

Schechner, Nayar, Generalized mosaics

)(Mask),(),(Raw xyxIyxk

2pair ingcorrespondeach

pixels all

2total

DIFFDIFF :Minimize

Bias towards “no motion”

Schechner, Nayar, Generalized mosaics

Image Registration

)(Mask),(),(Raw xyxIyxk

),( )(Mask

),(RawyxI

x

yxk

k

Raw

Mask

Uncertainty

),( yxIk

1

22

22 1 where ,

k kk k

k

II

I

III

2

22

2

2

21

12pair pixel

III

III

DIFF

Schechner, Nayar, Generalized mosaics

Image Registration

Registration: Standard “Coarse to Fine”

5 pixels

10 pix 9 pix

18 pix 19 pix

4 pixels

2 pixelsmotion of

However, we need: multiscale UNCERTAINTIES 2

22

2

2

21

12pair pixel

III

III

DIFF

Schechner, Nayar, Generalized mosaics

Maximum-Likelihood Pyramid At each level L , for each pixel

LL II , estimate

,

),( ),(yx

yxIyxwI

,

),( yx

yxw

,

),( 2 yx

yxwI-1

),(

),(),(yxI

yxayxw

2

Schechner, Nayar, Generalized mosaics

Max-Likelihood Pyramid

Gaussian Pyramid1

22

22

)(

1

)(

)( ,

xx xII

xI

xIII

x

)(xg

)1(I )2(I )3(I )4(I )5(I

x

xIxgI )()(

x )1(I )2(I )3(I )4(I )5(I )1(I )2(I )3(I )4(I )5(I

ML estimation

x

xIxnI )()(

)(

)()(2 xI

xgx

1][ )(

xn

1

2 )(

x

xI

log I

Inte

nsit

y ra

nge

Spatial range x

Generalized Mosaic

x

log I

Inte

nsit

y ra

nge

Spatial range

frame

fram

e

Mosaic

Schechner, Nayar, Generalized mosaics

Generalized Mosaicing: Yoav Schechner and Shree Nayar

spectral

y

x

Variable Spectral Filter

MMuullttii SSppeeccttrraall Mosaic

500 700400 600 500 700400 600 500 700400 600

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Rendering : Any Illumination

sunset fluorescent

Halogeny

x

Illumination at a Glance

Fluorescenty

x

Illuminant spectrum

Extra information

Spatially Varying Polarizer

Polarization Mosaicing: Yoav Schechner and Shree Nayar

0

0.2

0.4

0.6

0.8

1

-30

60

90

30

0

-60

-90

o

o

o

o

o

o

polarizance

polarizing angle

o

Polarization Mosaicing: Yoav Schechner and Shree Nayar

transmittance

Spatially Varying Polarizer

Raw images

Polarization Mosaicing: Yoav Schechner and Shree Nayar

Polarization Mosaic

Schechner Shamir Kiryati

JOSA-A 2000

reflected structure

transmitted painting

=

y

xDepth

Insert a prism behind the lens

Spatially Varying Focus

Generalized Mosaicing: Yoav Schechner and Shree Nayar

All FocusedAll Focused Mosaic

Generalized Mosaicing: Yoav Schechner and Shree Nayar

… and a Depth Map

What else?

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Dynamic Range & Spectrum & Polarization etc.

Multi-dimensional Mosaics - Simultaneously

Signal undergoing LPF

“Band limited” signal

Minimizing aliasing

Nyquist sampling rate

111111111111111

1?10111001??

Sampling Criteria

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Number of samples per scene point

resolution

length Lbandwidth B

aperture D

22 )/()/(

2

BLD

M2

log11 M

1011100101

1011100???

M # samples

1 11/2 21/4 3

92561

101110????

MosaicGeneralized Mosaic

Spatially varying filter

Wide field of view

SpectrumI

Depth + Focus

Dynamic range88 – 18,794

Polarization