High-performance imaging using dense arrays of cameras
Transcript of High-performance imaging using dense arrays of cameras
![Page 1: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/1.jpg)
Light Field
![Page 2: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/2.jpg)
Modeling a desktop
![Page 3: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/3.jpg)
![Page 4: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/4.jpg)
Image Based Rendering
Fast Realistic Rendering without 3D models
![Page 5: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/5.jpg)
Start from Ray Tracing
Rendering is about computing color along each ray
![Page 6: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/6.jpg)
Sampling Rays
![Page 7: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/7.jpg)
Sampling Rays by Taking Pictures
![Page 8: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/8.jpg)
Rendering as Ray Resampling
![Page 9: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/9.jpg)
Ray space
How to parameterize the ray space
How to sample and resample rays
![Page 10: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/10.jpg)
Two Plane Parameterization
![Page 11: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/11.jpg)
Stanford Camera Array
![Page 12: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/12.jpg)
Light Field Rendering
Very Fast
![Page 13: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/13.jpg)
Light Field Rendering
4D interpolation
![Page 14: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/14.jpg)
Dynamic Reparameterized Light Fields
![Page 15: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/15.jpg)
Dynamic Reparameterized Light Fields
• Move to desired new focal surface
• Create a new 4D space with new focal surface
• Recove ray with Reparameterization
• (u, v, s, t) => (u, v, f, g)F
![Page 16: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/16.jpg)
Dynamic Reparameterized Light Fields
• Recover ray r
• Resample from ray (s’, t’, f, g) and (s’’, t’’, f, g)
• Interpolation, reconstruction with filter, … , etc
![Page 17: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/17.jpg)
Dynamic Reparameterized Light Fields
• Change the shape of focal surface
• Gives focus on 3D object rather than planes
![Page 18: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/18.jpg)
Dynamic Reparameterized Light Fields
![Page 19: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/19.jpg)
Dynamic Reparameterized Light Fields
![Page 20: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/20.jpg)
Variable Apertures
• Also can generate variable aperture
• Aperture
– Control amount of light
– Control depth of fields
• Aperture Filter:
– Control how many cameras are used to resample a required ray
– Larger apertures produce images with narrow range of focus
![Page 21: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/21.jpg)
Aperture Filters
![Page 22: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/22.jpg)
Variable Apertures
![Page 23: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/23.jpg)
Variable Apertures
![Page 24: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/24.jpg)
Marc Levoy
Stanford multi-camera array
• 640 480 pixels
30 fps 128 cameras
• synchronized timing
• continuous streaming
• flexible arrangement
![Page 25: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/25.jpg)
Marc Levoy
Ways to use large camera arrays
• widely spaced light field capture
• tightly packed high-performance imaging
• intermediate spacing synthetic aperture photography
![Page 26: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/26.jpg)
Marc Levoy
Intermediate camera spacing:synthetic aperture photography
![Page 27: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/27.jpg)
Marc Levoy
Example using 45 cameras[Vaish CVPR 2004]
![Page 28: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/28.jpg)
![Page 29: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/29.jpg)
Tiled camera array
• world’s largest video camera
• no parallax for distant objects
• poor lenses limit image quality
• seamless mosaicing isn’t hard
Can we match the image quality of a cinema camera?
![Page 30: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/30.jpg)
Tiled panoramic image(before geometric or color calibration)
![Page 31: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/31.jpg)
Tiled panoramic image(after calibration and blending)
![Page 32: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/32.jpg)
Tiled camera array
• world’s largest video camera
• no parallax for distant objects
• poor lenses limit image quality
• seamless mosaicing isn’t hard
• per-camera exposure metering
• HDR within and between tiles
Can we match the image quality of a cinema camera?
![Page 33: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/33.jpg)
same exposure
in all cameras
individually
metered
checkerboard
of exposures
![Page 34: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/34.jpg)
Marc Levoy
High-performance photography as multi-dimensional sampling
• spatial resolution
• field of view
• frame rate
• dynamic range
• bits of precision
• depth of field
• focus setting
• color sensitivity
![Page 35: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/35.jpg)
Light field photography using a handheld plenoptic camera
Ren Ng, Marc Levoy, Mathieu Brédif,
Gene Duval, Mark Horowitz and Pat Hanrahan
Stanford University
![Page 36: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/36.jpg)
Marc Levoy
What’s wrong with conventional cameras?
Aperture
![Page 37: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/37.jpg)
Marc Levoy
Capture the light field inside a camera
125*125 μm
500 microns
Aperture
![Page 38: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/38.jpg)
Marc Levoy
Conventional versus light field camera
uv-plane st-plane
![Page 39: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/39.jpg)
Marc Levoy
Conventional versus light field camera
uv-planest-plane
![Page 40: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/40.jpg)
Prototype camera
4000 4000 pixels 292 292 lenses = 14 14 pixels per lens
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses
![Page 41: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/41.jpg)
Marc Levoy
Light Field in a Single Exposure
![Page 42: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/42.jpg)
Marc Levoy
Light Field in a Single Exposure
![Page 43: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/43.jpg)
Marc Levoy
Light field inside a camera body
![Page 44: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/44.jpg)
Marc Levoy
Digitally stopping-down
• stopping down = summing only the central portion of each microlens
Σ
Σ
![Page 45: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/45.jpg)
Marc Levoy
Digital refocusing
• refocusing = summing windows extracted from several microlenses
Σ
Σ
![Page 46: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/46.jpg)
Marc Levoy
Example of digital refocusing
![Page 47: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/47.jpg)
Marc Levoy
Refocusing portraits
![Page 48: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/48.jpg)
Marc Levoy
Action photography
![Page 49: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/49.jpg)
Extending the depth of field
conventional photograph,
main lens at f / 22
conventional photograph,
main lens at f / 4
![Page 50: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/50.jpg)
Marc Levoy
Scene-dependent focal plane
Σ
Depth from focus problem
Interactive solution [Agarwala 2004]
![Page 51: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/51.jpg)
Marc Levoy
Extending the depth of field
conventional photograph,
main lens at f / 22
conventional photograph,
main lens at f / 4
light field, main lens at f / 4,
after all-focus algorithm[Agarwala 2004]
![Page 52: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/52.jpg)
Marc Levoy
A digital refocusing theorem
• an f / N light field camera, with P P pixels under each microlens, can produce views as sharp as an f / (N P) conventional camera
• these views can be focused anywhere within the depth of field of the f / (N P) camera
![Page 53: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/53.jpg)
Marc Levoy
Prior work
• integral photography
– microlens array + film
– application is autostereoscopic effect
• [Adelson 1992]
– proposed this camera
– built an optical bench prototype using relay lenses
– application was stereo vision, not photography
![Page 54: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/54.jpg)
Marc Levoy
Digitally moving the observer
• moving the observer = moving the window we extract from the microlenses
Σ
Σ
![Page 55: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/55.jpg)
Marc Levoy
Example of moving the observer
![Page 56: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/56.jpg)
Marc Levoy
Moving backward and forward
![Page 57: High-performance imaging using dense arrays of cameras](https://reader030.fdocuments.us/reader030/viewer/2022012801/61bd091261276e740b0eb164/html5/thumbnails/57.jpg)
Marc Levoy
Implications
• cuts the unwanted link between exposure(due to the aperture) and depth of field
• trades off (excess) spatial resolution for ability to refocus and adjust the perspective
• sensor pixels should be made even smaller, subject to the diffraction limit
36mm 24mm 2.5μ pixels = 266 megapixels
20K 13K pixels
4000 2666 pixels 20 20 rays per pixel
• Application in microscope