and Computational Photography - … · What is Computational Photography? Computational photography...

Pixels, Numbers, and ProgramsComputational PhotographyComputational Photography

Steven L. Tanimoto

Pixels, Numbers, and Programs; © S. Tanimoto ‐‐ 1Computational Photography

Outline

Wh t i t ti l h t h ?What is computational photography?Autofocus techniquesHi h d i (HDR) h hHigh dynamic‐range (HDR) photographyCatadioptric camerasSeparating images due to direct and indirect lightingOther developments

Pixels, Numbers, and Programs; © S. Tanimoto ‐‐ Computational Photography 2

What is Computational Photography?

Computational photography is an emerging technology forComputational photography is an emerging technology for acquiring images through a combination of optics, sensors, and computers.

It includes automatic camera control, computation of pixel values using multiple sources of information, as well as g p ,computing alternative representations of visual information to 2D images (e.g., 3D models).


Autofocus techniques

Focusing a camera lens was once a tedious chore.Focusing a camera lens was once a tedious chore.Instant (film) cameras used fixed focus, limiting photography to well‐lit scenes and fast (high ASA) p g p y ( g )films.Acoustic rangefinding was used on some cameras to g f gdetermine the distance from the camera to the first surface in the middle of the field of view.Digital cameras change the economics of autofocus. Now, techniques based on sharpness measurements

ff tiPixels, Numbers, and Programs; © S. Tanimoto ‐‐ Computational Photography

are effective.4

Autofocus techniques (cont.)

Brenner sharpness:

Tenengrad sharpness:

Here Gx and Gy are the horizontal and vertical gradient operators

Pixels, Numbers, and Programs; © S. Tanimoto ‐‐ Computational Photography

x y g pused in the Sobel edge detector.

5

Autofocus techniques (cont.)

x

Frame #1 Frame #10

Sharpness as a function of focus setting. Best focus


gis at the 10th frame.

High Dynamic‐Range Photography

The dynamic range of a photo or camera is the rangeThe dynamic range of a photo or camera is the range of brightness levels that it can accurately capture.Historically, dynamic range was limited by (a) film y, y g y ( )technology, and (b) digital light sensor technology.

The photographer’s challenge: take a picture of a scene that contains both bright and dark regions, without losing the details in one of these regions.


HDR (cont.)

The brightly lit area is capturedThe brightly lit area is captured effectively in this photo (taken with a 1/60 sec. exposure).

But the lower half of the photo p(the indoor part of the scene) is too dark and lacking in contrast.


HDR (cont.)

Here, the darkest areas areHere, the darkest areas are captured effectively (taken with a 15 sec. exposure).

But the upper half of the photo pp p(the outdoor part of the scene) is too bright and lacking in contrast.


HDR (cont.)

Now, both areas have someNow, both areas have some visible structure (taken with a 1/2 sec. exposure).

But neither shows details well.


HDR (cont.)

HDR to the rescue!HDR to the rescue!Taking a full sequence of exposures, we build a “high‐p , gdynamic range” image inside the computer. From it, we apply “tone mapping” to get an image that has good contrast across the rangerange.


HDR (cont.)

HDR to the rescue!HDR to the rescue!Taking a full sequence of exposures, we build a “high‐p , gdynamic range” image inside the computer. From it, we apply “tone mapping” to get an image that has good contrast across the range Th id l i i i frange. The idea: get luminosity info about each pixel from each image, but weight the evidence according to what each image is good at


good at.

12

Catadioptric Imaging

Using optical systems that combine lenses andUsing optical systems that combine lenses and mirrors, anamorphic images can be captured that contain entire panoramas.Computers can invert the anamorphic distortion.

Courtesy of Prof. Shree Nayar of Columbia University.


Catadioptric Imaging


Catadioptric Cameras

The One‐shot 360 (RemoteReality, Inc.). Three other catadioptric cameras. http://www.cs.columbia.edu/CAVE/projects/cat_cam_360/


Separating Direct & Indirect Lighting

The color of an object (specifically at a point P on the object) in a scene depends on two different physical effects:depends on two different physical effects:a. Direct reflection of light from the light source off the surface of the

object (at point P), andb Light coming from point P on the object that did not come directlyb. Light coming from point P on the object that did not come directly

from the light source:i. light that has come from the light source but bounced off other

surfaces before getting to Psurfaces before getting to P.ii. light that has entered the surface material (e.g., the paint layer,

which is typically a dielectric material and is partially translucent) elsewhere from point P but emerges from point Pelsewhere from point P but emerges from point P.

We are often unaware of (ii). However, using computational photography, we can separate the effects of (i) and (ii) The results are amazing


we can separate the effects of (i) and (ii). The results are amazing.

16

Direct & Indirect (cont.)

indirectlight source surfaces in

the scene

P

camera

Rt

Radiance at P towards the camera: Rt = Rd + RgRg is the “global” component of radiance (based on all indirect illumination of P).Rt is the total radiance.


Direct & Indirect (cont.)

indirectlight source

Barrier to cast a shadow on P

P

camera

Rg

Radiance at P towards the camera: Rt = Rd + RgRg is the “global” component of radiance (based on all indirect illumination of P).Rt is the total radiance.By measuring R and R then we can compute R as R R


By measuring Rt and Rg then we can compute Rd as Rt Rg

Results of Separation

By using structured light (projection of checkerboards), it is not necessary to have a separate shadow image for each pixel. In theory, only 2 imagesto have a separate shadow image for each pixel. In theory, only 2 images are needed, but in practice it’s best to have 10 to 20 images, according to Krishnan and Nayar.

Rt Rd Rg

In this example most of the radiance from global illumination is due toIn this example, most of the radiance from global illumination is due to interreflectance among the eggs.


Results of Separation

A scene with grapes and cheese…Much of the natural color (as seen in R ) comes from R and not RMuch of the natural color (as seen in Rt ) comes from Rg and not Rd .

Rt Rd Rg

Here, much of the radiance from global illumination is due to light passing through the translucent grapes or cheese.The cheese, shown with direct illumination only, is somewhat unappetizing. The grapes also appear blue.


Applications: image enhancement, image analysis.20

Other Developments

•3D cameras – camera takes two or more shots of same scene, then builds 3D model of scenethen builds 3D model of scene.•Redeye reduction. Already standard is the automatic use of a pre‐flash to condition the pupils of photo subjects to close down. Redeye can also be reduced using face‐detection methods and specialized algorithms.•Light field cameras By capturing a sample of the complete light•Light‐field cameras. By capturing a sample of the complete light field at the viewpoint, operations such as focusing on specific objects can be performed later.


Other Developments (cont.)

•In‐camera HDR. Automatic exposure bracketing is leading the way to built in HDR captureway to built‐in HDR capture.•Motion blur inversion. The camera can estimate its motion using a combination of accelerometer data and video data. The motion info can then be used to partially invert the motion blur.•All smiles snapshots. Using face‐detection, smile‐detection algorithms and inpainting techniques it is possibly to synthesizealgorithms, and inpainting techniques, it is possibly to synthesize an image of a group of people in which everyone is smiling, even though in each component shot, there was someone not smiling.


and Computational Photography - … · What is Computational Photography? Computational photography...

Documents

Transcript of and Computational Photography - … · What is Computational Photography? Computational photography...