Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon...

72
Coded Computational Photography Gordon Wetzstein Stanford University EE367/CS448I: Computational Imaging and Display stanford.edu/class/ee367 Lecture 9

Transcript of Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon...

Page 1: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Computational Photography!

Gordon Wetzstein!Stanford University!

EE367/CS448I: Computational Imaging and Display!stanford.edu/class/ee367!

Lecture 9!

Page 2: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Computational Photography - Overview!•! coded apertures!•! extended depth of field!

•! wavefront coding!•! lattice lens!•! diffusion coding!

•! focal sweep!•! motion deblurring !

•! flutter shutter!•! motion invariance!

[Ras

kar e

t al.

2006

]![C

ossa

irt e

t al.,

201

0]!

Page 3: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Remember Apertures?!•! out of focus blur!

focal plane!

circ

le o

f con

fusi

on!

circ

le o

f con

fusi

on

Page 4: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

What makes Defocus Deblurring Hard?!1.! depth-dependent PSF scale (depth unknown)!2.! circular / Airy PSF is not (well) invertible!

focal plane!

circ

le o

f con

fusi

on!

circ

le o

f con

fusi

on

Page 5: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Computational Imaging - Motivation !1.  depth-dependent PSF scale (depth unknown)!

•  engineer PSF to be depth invariant!

•  resulting shift-invariant deconvolution is much easier!!

2.  circular / Airy PSF is not (well) invertible: ill-posed problem!

•  engineer PSF to be broadband (flat Fourier magnitudes)!•  resulting inverse problem becomes well-posed!

Page 6: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Computational Imaging!

?

?

?

•! new optics!•! new sensors!

•! new illumination!

•! new algorithms!

????

1.! optically encode scene information!2.! computationally recover information!

Page 7: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Computational Imaging (for this Class)!

?

•! new optics!•! easier algorithms!!

?

1.! optically encode scene information using invertible (and possibly invariant) PSF !

2.! computationally recover information (easy

because of engineered PSF)!

Page 8: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Computational Imaging (for this Class)!

?

•! new optics!•! easier algorithms!!

?

idea applies to !•! coded apertures!

•! extended depth of field / DOF deblurring!

•! extended motion / motion deblurring!

Page 9: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Before going to Advanced Techniques for DOF Deblurring, let’s take a look at !

Coded Apertures!

Page 10: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Apertures Revisited!•! two important parts:!

1.! aperture stop – attenuating pattern!

2.! refractive element (lens or compound lens system)!

1. attenuating coded aperture: e.g., MURA pattern!

2. refractive coded!aperture: e.g., cubic phase plate!

Page 11: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Aperture Changes PSF!

in-focus photo! out-of-focus, circular aperture! out-of-focus, coded aperture!

[Veeraragharavan et al. 2007]!

Page 12: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Aperture Changes PSF!

in-focus photo! out-of-focus, circular aperture! out-of-focus, coded aperture!

[Veeraragharavan et al. 2007]!

Page 13: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Aperture Changes PSF!

•! preserves high frequencies!•! deconvolution well-posed!!

FFT!

[Veeraragharavan et al. 2007]!

coded!

conventional!

Page 14: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Aperture Allows for Depth Estimation!•! introduce zeros in Fourier domain!•! better depth discimination!

•! worse invertibility!

conventional aperture! coded aperture! PSF! [Levin et al. 2007]!

Page 15: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Coded Aperture Allows for Depth Estimation!•! deconvolution with strong prior necessary!

input! local depth estimate! regularized depth![Levin et al. 2007]!

Page 16: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

In Astronomy!•! some wavelengths are difficult to focus!! no “lenses” available!

•! coded apertures for x-rays and gamma rays!

NASA Swift!

ESA SPI / INTEGRAL!

Page 17: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

In Microscopy!•! for low-light, coding of refraction is better (less light loss)!

e.g., cubic phase plate for depth-invariant imaging!

e.g., rotating double helix PSF Stanford Moerner lab!

Page 18: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Extended Depth of Field!

Page 19: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Depth Invariant PSFs - Overview!•! two general approaches:!

1.! move sensor / object!(known as focal sweep)!

2. change optics!(e.g., wavefront coding)!

Page 20: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep!

[Nagahara et al. 2008]!

time!

dist

ance

!se

nsor

-lens

!

time!

dist

ance

!se

nsor

-lens

!

time!

dist

ance

!se

nsor

-lens

!

linear motion:!

nonlinear motion:!

nonlinear motion:!

exposure!

Page 21: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep! [Nagahara et al. 2008]!

time!

dist

ance

!se

nsor

-lens

!

time!

two points at different distance !

time

Page 22: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep! [Nagahara et al. 2008]!

time!

dist

ance

!se

nsor

-lens

!

time!

!!

PSF 1!

PSF 2!

t1 t2

instantaneous PSF! integrated PSF!integrated PSFinstantaneous PSF

two points at different distance !

timetime

Page 23: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep! [Nagahara et al. 2008]!

time!

dist

ance

!se

nsor

-lens

!

time!

!!

PSF 1!

PSF 2!

t1 t2

instantaneous PSF!instantaneous PSF

two points at different distance !

timetime

Page 24: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep! [Nagahara et al. 2008]!

time!

dist

ance

!se

nsor

-lens

!

time!

!!

PSF 1!

PSF 2!

t1 t3t2

instantaneous PSF! integrated PSF!instantaneous PSF

two points at different distance !

timetime

Page 25: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep! [Nagahara et al. 2008]!

time!

dist

ance

!se

nsor

-lens

!

time!

!!

PSF 1!

PSF 2!

t1 t3t2 t4

instantaneous PSF!instantaneous PSF

two points at different distance !

timetime

Page 26: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep! [Nagahara et al. 2008]!

time!

dist

ance

!se

nsor

-lens

!

time!

! dt =

! dt = !!

PSF 1!

PSF 2!

t1 t3t2 t4 t5

instantaneous PSF! integrated PSF!integrated PSFinstantaneous PSF

two points at different distance !

timetime

Page 27: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep! [Nagahara et al. 2008]!

time!

dist

ance

!se

nsor

-lens

!

time!

! dt =

! dt = !!

PSF 1!

PSF 2!

t1 t3t2 t4 t5

instantaneous PSF! integrated PSF!integrated PSFinstantaneous PSF

two points at different distance !

timetime

Page 28: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep! [Nagahara et al. 2008]!

time!

dist

ance

!se

nsor

-lens

!

time!

!!

integrated PSF!integrated PSF

two points at different distance !

timetime

•! spend equal amount of time at each depth to make depth invariant!!

!

• spend equal amount of time at each depth to make depth invariant!

Page 29: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep![N

agah

ara

et a

l. 20

08]!

Focal SweepFocal SweepFocal Sweepco

nven

tiona

l pho

to

(sm

all D

OF)!

capt

ured

foca

l sw

eep!

alw

ays

blur

ry!!

conv

entio

nal p

hoto

(la

rge

DO

F, n

oisy

)!ED

OF

imag

e!

Page 30: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep![N

agah

ara

et a

l. 20

08]!

Focal Sweep

conv

entio

nal p

hoto

(la

rge

DO

F, n

oisy

)!ED

OF

imag

e!

•! noise characteristics are main benefit of EDOF!

•! may change for different sensor

noise characteristics!

!

SNR should be!evaluation metric!!

Page 31: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep for Moving Objects!

motion!

conventional camera PSF! focal sweep camera PSF!

motion motion!de

focu

s!de

focu

s

[Bando et al. 2013]!

Page 32: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep for Moving Objects!

motion!

conventional camera PSF! focal sweep camera PSF!

motion motion!de

focu

s!de

focu

s

[Bando et al. 2013]!

Page 33: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep for Moving Objects!

motion!

conventional camera PSF! focal sweep camera PSF!

motion motion!de

focu

s!de

focu

s

[Bando et al. 2013]!

Page 34: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Focal Sweep for Moving Objects!

[Bando et al. 2013]!

conventional camera! focal sweep! focal sweep deblurred!

Page 35: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Wavefront Coding!•! how to obtain a depth invariant PSF without mechanically moving parts!

! change the lens!!

•! for many, this is the dawn of computational imaging!!

!

[Dowski and Cathey 1995]!

cubic phase plate!

•! tricky to understand intuitively, so let’s try to understand what it does

by looking at something else…!

!

Page 36: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Lattice Focal Lens!

[Levin et al. 2009]!

superimpose array of lenses with different focal lengths!!

time!timetimetime

Page 37: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Lattice Focal Lens!

[Levin et al. 2009]!

conventional camera! lattice focal lens! all-in-focus image from lattice focal lens!

Page 38: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Extended Depth of Field (EDOF)!

•  remember focal sweep: move sensor s.t. same time for each depth!

•  lattice focal lens: same idea, but no sweeping (optical overlay) –

optimal in 4D!

•  cubic phase plate: same idea (optimal in 2D, not optimal in 4D)!(can look at this in more detail if we have time)!

Page 39: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Diffusion Coded Photography!•! can also do EDOF with diffuser as coded aperture, has better inversion !

characteristics than lattice focal lens!

[Cossairt et al. 2010]!

Page 40: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Back to Coding Motion!

Page 41: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Flutter Shutter!•  engineer motion PSF (coding exposure time) so it becomes invertible!!

[Raskar et al. 2006]!

Page 42: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

[Raskar et al. 2006]!photo with coded motion!

Page 43: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

deblurred!

Page 44: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Deblurred Result!Input Photo!

[Raskar et al. 2006]!

Page 45: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Traditional Camera!!

Shutter is OPEN!

[Raskar et al. 2006]!

Page 46: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

!Flutter Shutter!

[Raskar et al. 2006]!

Page 47: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Shutter is OPEN and CLOSED!

!!

[Raskar et al. 2006]!

Page 48: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

H H

Har

old

“Doc

” Edg

erto

n

Page 49: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

[Raskar et al. 2006]!

Page 50: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Lab Setup [Raskar et al. 2006]!

Page 51: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Blurring !=!

Convolution!

Traditional Camera: Box Filter!

sinc Function!

[Raskar et al. 2006]!

Fourier magnitudes!

spatial convolution!

Page 52: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Flutter Shutter: Coded Filter!

Preserves High Frequencies!!!!

[Raskar et al. 2006]!

spatial convolution!

Fourier magnitudes!

Page 53: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Comparison! [Raskar et al. 2006]!

Page 54: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Inverse Filter Unstable!

Inverse Filter stable!

[Raskar et al. 2006]!

Page 55: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Short Exposure Long Exposure Coded Exposure

Ground Truth Matlab Richardson-Lucy

Our result

Page 56: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Are all codes good ?!

Alternate!

All ones!

Random!

Our Code!

[Raskar et al. 2006]!

Page 57: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

License Plate Retrieval!

[Raskar et al. 2006]!

Page 58: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

License Plate Retrieval!

[Raskar et al. 2006]!

Page 59: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Motion Invariant Photography!

•  making motion PSFs invariant is great, BUT need to know motion direction and velocity!!

•  we have already seen that focal sweep makes the PSF almost depth

invariant!

!•  how about making motion PSFs motion invariant?!

Page 60: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

title!•! text!

titletext

Jacq

ues

Hen

ri La

rtigu

e, 1

912!

Page 61: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

•  text!an

imat

ion

by l

arge

form

atph

otog

raph

y.in

fo u

ser L

indo

lfi!

Page 62: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Controlling Motion Blur! [Levin et al. 2008]!

Page 63: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Can we control motion blur?!

Controlling Motion Blur! [Levin et al. 2008]!

Page 64: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Controlling Motion Blur! [Levin et al. 2008]!

Page 65: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Controlling Motion Blur! [Levin et al. 2008]!

Page 66: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Motion invariant blur

Controlling Motion Blur! [Levin et al. 2008]!

?!

Page 67: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Sensor position x(t)=a t2!

•  start by moving very fast to the right!•  continuously slow down until stop!

•  continuously accelerate to the left!!•  Intuition:!

•  for any velocity, there is one instant where we track perfectly!

•  all velocities captured same amount of time!

!Sensor position x!

Time t!

Parabolic Sweep! [Levin et al. 2008]!

Page 68: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Motion Invariant Blur! [Levin et al. 2008]!

Page 69: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Static camera!!

Unknown and variable blur kernels!

Our parabolic input!!

Blur kernel is invariant to velocity!

Our output after deblurring!

NON-BLIND deconvolution!

!

[Levin et al. 2008]!

Page 70: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Equal high response in all range!

Primal Domain!t!

x!

Frequency Domain!

t!

x!

t!

x!

Objects!

Camera integration

curve!

Parabolic sweep!

t!

x!

Velocity 2!

Velocity 1!

Static!

Frequency Domain! Primal Domain Primal Domain Primal Domain Primal Domain Primal Domain Primal Domain Primal Domain Primal Domain Primal Domain

sensor integration!

[Levin et al. 2008]!

Page 71: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

Next: Noise!

•! Gaussian noise!•! Poissonian noise!

•! Denoising!

Page 72: Coded Computational Photography - web.stanford.edu · Coded Computational Photography! Gordon Wetzstein! Stanford University! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367!

References and Further Reading!Extended Depth of Field (EDOF)!•  DOWSKI, E. R., AND CATHEY, W. T. 1995. Extended depth of field through wave-front coding. Appl. Opt. 34, 11, 1859–1866!•  Levin, Hasinoff, Green, Durand, Freeman, “4D Frequency Analysis of Computational Cameras for Depth of Field Extension”, ACM SIGGRAPH 2009!•  Cossairt, Zhou, Nayar, “Diffusion-Coded Photography”, ACM SIGGRAPH 2012!•  overview and analysis in light field space: Zhang, Levoy, “Wigner Distributions and How They Relate to the Light Field”, ICCP 2009!•  A. Isaksen, L. McMillan, and S. J. Gortler. “Dynamically reparameterized light fields”. In Proc. ACM SIGGRAPH, 2000!

EDOF through Focal Sweep!•  HAUSLER , G. 1972. A method to increase the depth of focus by two step image processing. Optics Communications 6 (Sep), 38–42.!•  NAGAHARA, H., KUTHIRUMMAL, S., ZHOU, C., AND NAYAR, S. 2008. Flexible Depth of Field Photography. In ECCV ’08, 73!•  Cossairt, Nayar “Spectral Focal Sweep for Extending Depth of Field”, Proc. ICCP 2010!

Coded Apertures!•  LEVIN, A., FERGUS, R., DURAND, F., AND FREEMAN, W. T. 2007. Image and depth from a conventional camera with a coded aperture. In SIGGRAPH ’07, 70.!•  VEERARAGHAVAN, A., RASKAR, R., AGRAWAL, A., MOHAN, A., AND TUMBLIN, J. 2007. Dappled photography: mask enhanced cameras for heterodyned light

fields and coded aperture refocusing. In SIGGRAPH ’07, 69!•  ZHOU, C., AND NAYAR, S. 2009. What are Good Apertures for Defocus Deblurring? In ICCP ’09!

Coding Motion!•  Raskar, Agrawal, Tumblin, “Coded Exposure Photography: Motion Deblurring using Fluttered Shutter”, ACM SIGGRAPH 2006!•  Levin, Sand, Cho, Durand, Freeman, “Motion-Invariant Photography”, ACM SIGGRAPH 2008!

Motion and Depth Invariance!•  Bando, Holtzman, Raskar, “Near-Invariant Blur for Depth and 2D Motion via Time-Varying Light Field Analysis”, ACM Trans. Graph. 2013!•  Bando, “An Analysis of Focus Sweep for Improved 2D Motion Invariance”, IEEE CVPR CCD Workshop 2013!!