Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department...

59
Light field sensing Marc Levoy Computer Science Department Stanford University

Transcript of Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department...

Page 1: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Light field sensing

Marc Levoy

Computer Science Department Stanford University

Page 2: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

The scalar light field (in geometrical optics)

Radiance as a function of position and direction in a static scene with fixed illumination

L is radiance in watts / (m2 steradians) 5-dimensional function

Page 3: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

The vector light field [Gershun 1936]

•  amplitude gives irradiance at that point

•  direction tells which way to orient a surface for maximum brightness under uniform illumination

adding two light vectors the vector light field produced by a luminous strip

Page 4: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Visualizing the vector irradiance field

flatland scene with partially opaque blockers

under uniform illumination

scalar irradiance at each point

vector directions,visualized using line integral convolution (LIC) [Cabral 1993]

Page 5: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Dimensionality of the scalar light field

•  for general scenes ⇒ 5D function “plenoptic function”

L ( x, y, z, θ, φ )

•  in free space ⇒ 4D function “ the (scalar) light field”

L ( ? )

Page 6: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Some candidate parameterizations for the 4D light field

Point-on-plane + direction (or point-on-surface + direction)

L ( x, y, θ, φ ) or L ( u, v, θ, φ )

• convenient for measuring BRDFs • restriction to line gives looming field

x

y

Page 7: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

• 

flight path through a flatland scene

corresponding looming light field (see also [Hasinoff 2006])

θ

t

t

θ

Page 8: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

The looming field

[Gibson]

Page 9: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

More parameterizations

Chords of a sphere L ( θ1, φ1, θ2, φ2 )

•  convenient for spherical gantry •  facilitates uniform sampling

Page 10: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Two planes (“light slab”)

L ( u, v, s, t )

• uses projective geometry – one plane at infinity ⇒ array of orthographic images –  fast incremental display algorithms

Page 11: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

The free-space assumption

• Where can you use free-space light fields? –  the 3D space around a compact object –  the 3D space inside an uncluttered environment

•  stitching together light fields [Chen, Levoy, Hanrahan (unpublished) ]

– partition scene into disjoint cells –  links between cells are light fields – hierarchy of cells, links, light fields – # of light fields is linear in # of cells

Page 12: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Light field rendering

flipbook animation (QuickTime VR)

rebinning the rays to create new views

(movie is available at http://graphics.stanford.edu/papers/light)

Page 13: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Alternative parameterizations for the 5D plenoptic function

• Two-plane ray field

L ( u, v, s, t, z )

•  allows multiple colors, in sequence, along one line •  alternative to L ( x, y, z, θ, φ ) •  inspired by Salesin’s ZZ-buffer [1989]

z

Page 14: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Devices for recording light fields

small scenes

big scenes •  handheld camera [Buehler 2001]

•  array of cameras [Wilburn 2005] •  plenoptic camera [Ng 2005] •  light field microscope [Levoy 2006]

Page 15: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Stanford Multi-Camera Array [Wilburn SIGGRAPH 2005]

•  640 × 480 pixels × 30 fps × 128 cameras

•  synchronized timing •  continuous streaming •  flexible arrangement

Page 16: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2005 Marc Levoy

Ways to use large camera arrays

•  widely spaced light field capture

Manex’s bullet time array

(movie is available at http://graphics.stanford.edu/projects/array)

Page 17: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2005 Marc Levoy

Ways to use large camera arrays

•  widely spaced light field capture •  tightly packed high-performance imaging

Page 18: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2005 Marc Levoy

Ways to use large camera arrays

•  widely spaced light field capture •  tightly packed high-performance imaging •  intermediate spacing synthetic aperture photography

Page 19: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Synthetic aperture photography

Σ

Page 20: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Example using 45 cameras [Vaish CVPR 2004]

Page 21: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

one camera’s view synthetic aperture view

(movie is available at http://graphics.stanford.edu/projects/array)

Page 22: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Light field photography using a handheld plenoptic camera

Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan

(Proc. SIGGRAPH 2005 and TR 2005-02)

Page 23: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Conventional versus light field camera

Page 24: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Conventional versus light field camera

uv-plane st-plane

Page 25: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Prototype camera

4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Contax medium format camera Kodak 16-megapixel sensor

Adaptive Optics microlens array 125µ square-sided microlenses

Page 26: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Typical image captured by camera (show here at low res)

Page 27: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Digital refocusing

•  refocusing = summing windows extracted from several microlenses

Σ

Σ

Page 28: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2006 Marc Levoy

A digital refocusing theorem

•  an f / N light field camera, with P × P pixels under each microlens, can produce views as sharp as an f / (N × P) conventional camera

– or –

•  it can produce views with a shallow depth of field ( f / N ) focused anywhere within the depth of field of an f / (N × P) camera

Page 29: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2007 Marc Levoy

Example of digital refocusing

Page 30: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2007 Marc Levoy

Example of digital refocusing

Page 31: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2007 Marc Levoy

Example of digital refocusing

Page 32: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2007 Marc Levoy

Example of digital refocusing

Page 33: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2007 Marc Levoy

Example of digital refocusing

Page 34: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2007 Marc Levoy

Refocusing portraits

(movie is available at http://refocusimaging.com)

Page 35: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Extending the depth of field

conventional photograph, main lens at f / 22

conventional photograph, main lens at f / 4

light field, main lens at f / 4, after all-focus algorithm [Agarwala 2004]

Page 36: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Digitally moving the observer

•  moving the observer = moving the window we extract from the microlenses

Σ

Σ

Page 37: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Example of moving the observer

Page 38: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Example of moving the observer

Page 39: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Example of moving the observer

Page 40: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Moving backward and forward

Page 41: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Moving backward and forward

Page 42: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Moving backward and forward

Page 43: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Moving backward and forward

Page 44: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Lego gantry for capturing light fields (built by Andrew Adams)

calibration point

plane + parallax [Vaish 2004]

Page 45: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Flash-based viewer for light fields (written by Andrew Adams)

(light field can be viewed at http://lightfield.stanford.edu/lfs.html)

Page 46: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Implications / commercialization (see refocusimaging.com)

•  cuts the unwanted link between exposure (due to the aperture) and depth of field

•  trades off (excess) spatial resolution for ability to refocus and adjust the perspective

•  sensor pixels should be made even smaller, subject to the diffraction limit

36mm × 24mm ÷ 2.5µ pixels = 266 Mpix 20K × 13K pixels 4000 × 2666 pixels × 20 × 20 rays per pixel

or 2000 × 1500 pixels × 3 × 3 rays per pixel = 27 Mpix

Page 47: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Light Field Microscopy

Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz

(Proc. SIGGRAPH 2006)

Page 48: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

A traditional microscope

objective

specimen

intermediate image plane

eyepiece

Page 49: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

A light field microscope (LFM)

•  40x / 0.95NA objective ↓ 0.26µ spot on specimen × 40x = 10.4µ on sensor

↓ 2400 spots over 25mm field

•  1252-micron microlenses ↓ 200 × 200 microlenses with 12 × 12 spots per microlens

objective

specimen

intermediate image plane

eyepiece sensor

→ reduced lateral resolution on specimen = 0.26µ × 12 spots = 3.1µ

Page 50: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

A light field microscope (LFM)

objective

specimen

intermediate image plane

eyepiece sensor

Page 51: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

Example light field micrograph

•  orange fluorescent crayon •  mercury-arc source + blue dichroic filter •  16x / 0.5NA (dry) objective •  f/20 microlens array •  65mm f/2.8 macro lens at 1:1 •  Canon 20D digital camera

ordinary microscope light field microscope

Page 52: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2006 Marc Levoy

The geometry of the light field in a microscope

•  microscopes make orthographic views

•  translating the stage in X or Y provides no parallax on the specimen

•  out-of-plane features don’t shift position when they come into focus

•  front lens element size = aperture width + field width

•  PSF for 3D deconvolution microscopy is shift-invariant (i.e. doesn’t change across the field of view)

f

objective lenses are telecentric

Page 53: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Example light field micrograph

panning sequence focal stack

(movies are available at http://graphics.stanford.edu/projects/lfmicroscope)

Page 54: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Real-time viewer

(movie is available at http://graphics.stanford.edu/projects/lfmicroscope/2007.html)

Page 55: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Other examples

fern spore (60x, autofluorescence)

Golgi-stained neurons (40x, transmitted light)

zebrafish optic tectum (calcium imaging of neural activity)

(movies are available at http://graphics.stanford.edu/projects/lfmicroscope)

Page 56: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

3D reconstruction

•  4D light field → digital refocusing → 3D focal stack → deconvolution microscopy → 3D volume data

•  4D light field → tomographic reconstruction → 3D volume data

(from Kak & Slaney)

(DeltaVision)

Page 57: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

Silkworm mouth (40x / 1.3NA oil immersion)

slice of focal stack slice of volume volume rendering

100µ

Page 58: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2008 Marc Levoy

GFP-labeled zebrafish neurons (40x / 0.8NA water immersion)

focal stack

deconvolved

volume rendering

Page 59: Marc Levoy - Computer Graphics · Light field sensing Marc Levoy Computer Science Department Stanford University

© 2005 Marc Levoy http://graphics.stanford.edu