OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What...

23
OUTLINE OUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography: Perfecting Film-Like Photography: Old Problems, New Approaches Old Problems, New Approaches Photography Beyond Film: Photography Beyond Film: New Goals, Methods, Expressions New Goals, Methods, Expressions

Transcript of OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What...

Page 1: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

OUTLINEOUTLINEOUTLINEOUTLINE

• What is Photography? What is Photography?

• What is ‘The Photographic Signal’? What is ‘The Photographic Signal’?

• Perfecting Film-Like Photography: Perfecting Film-Like Photography: Old Problems, New ApproachesOld Problems, New Approaches

• Photography Beyond Film: Photography Beyond Film: New Goals, Methods, ExpressionsNew Goals, Methods, Expressions

Page 2: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Course 15:

Computational PhotographyCourse 15:Course 15:

Computational Photography

Page 3: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Jack TumblinJack Tumblin

Northwestern UniversityNorthwestern University

A2:A2: Core ConceptsCore Concepts(15 minutes)(15 minutes)

Computational PhotographyComputational Photography

Page 4: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

‘‘Film-Like’ PhotographyFilm-Like’ Photography‘‘Film-Like’ PhotographyFilm-Like’ Photography

Film Camera design assumptions:Film Camera design assumptions:

– ‘‘Instantaneous’ light measurement…Instantaneous’ light measurement…

– Of focal plane image behind a lens.Of focal plane image behind a lens.

– Reproduce those amounts of light.Reproduce those amounts of light.

Implied:Implied:

“ “What we see is What we see is focal-plane intensities.” focal-plane intensities.” well, no well, no…we see …we see muchmuch more! more!

(seeing is deeply cognitive)(seeing is deeply cognitive)

Page 5: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Our Definitions Our Definitions Our Definitions Our Definitions

• ‘‘Film-like’ Photography:Film-like’ Photography:

– Static ‘instantaneous’ record of Static ‘instantaneous’ record of the 2D image formed by a lensthe 2D image formed by a lens

Display image Display image sensor image sensor image

• ‘‘Computational’ Photography:Computational’ Photography:

– displayed image displayed image sensor image sensor image

– A more expressive, controllable displayed result,A more expressive, controllable displayed result,

from transformed, merged, decoded sensor datafrom transformed, merged, decoded sensor data

Page 6: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

What What isis Photography? Photography?What What isis Photography? Photography?

• A ‘bucket’ word: a neat container for messy notionsA ‘bucket’ word: a neat container for messy notions(e.g. aviation, music, comprehension)(e.g. aviation, music, comprehension)

• A record of what we see,A record of what we see,or would like to see,or would like to see,in tangible form.in tangible form.

• Does ‘film’ photography Does ‘film’ photography always capture it? always capture it? no.

• So, what do we see?So, what do we see?Harold ‘Doc’ Edgerton 1936Harold ‘Doc’ Edgerton 1936

Page 7: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

DisplayDisplayRGB(x,y,tRGB(x,y,tnn))

ImageImageI(x,y,I(x,y,λλ,t),t)

Light &Optics3D Scene3D Scene

light sources, BRDFs,shapes,

positions,movements,

EyepointEyepoint position,

movement,projection,

PHYSICALPHYSICAL PERCEIVEDPERCEIVED

What What isis Photography? Photography?

Exposure Exposure Control,Control,

tone maptone mapSceneScenelight sources,BRDFs,shapes,positions,movements,…

EyepointEyepointposition, movement,projection,…

Vis

ion

Photo: A Tangible RecordPhoto: A Tangible RecordEditable, storable asEditable, storable as

Film or PixelsFilm or Pixels

Page 8: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

3D Scene?3D Scene?light sources, BRDFs,shapes,positions,movements,…

Eyepoint?Eyepoint?position, movement,projection,…

Meaning…Meaning…

VisualVisualStimulusStimulus

3D Scene3D Scenelight sources,

BRDFs,shapes,

positions,movements,

EyepointEyepoint position,

movement,projection,

PHYSICALPHYSICAL PERCEIVED PERCEIVED or UNDERSTOODor UNDERSTOOD

Ultimate Photographic GoalsUltimate Photographic Goals

Vis

ion

Vis

ion

Sen

sor(

s)S

enso

r(s)

Com

putin

gC

ompu

ting

Light &Light &OpticsOptics

Photo: A Tangible RecordPhoto: A Tangible RecordScene Scene estimates we canestimates we can

capture, edit, store, displaycapture, edit, store, display

Page 9: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Missing:Missing: Viewpoint Freedom Viewpoint Freedom

Missing:Missing: Viewpoint Freedom Viewpoint Freedom

““Multiple-Center-of-Projection Images” Rademacher, P, Bishop, G., SIGGRAPH '98Multiple-Center-of-Projection Images” Rademacher, P, Bishop, G., SIGGRAPH '98

Page 10: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Missing: Missing: Reliable Visual BoundariesReliable Visual Boundaries

Missing: Missing: Reliable Visual BoundariesReliable Visual Boundaries

5 ray sets 5 ray sets explicit geometric occlusion boundaries explicit geometric occlusion boundaries

Ramesh Raskar, MERL, 2004

Page 11: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Missing:Missing: Expressive Time ManipulationsExpressive Time Manipulations

Missing:Missing: Expressive Time ManipulationsExpressive Time Manipulations

What other waysWhat other waysbetter better revealreveal appearanceappearance to to human viewers?human viewers?

(Without direct shape (Without direct shape measurement? )measurement? )

Time for space wiggle. Time for space wiggle. Gasparini, 1998.

Can you understandCan you understand this shape better?this shape better?

Page 12: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Photographic Signal: Pixels RaysPhotographic Signal: Pixels RaysPhotographic Signal: Pixels RaysPhotographic Signal: Pixels Rays

• Core ideas are ancient, simple, seem obvious:Core ideas are ancient, simple, seem obvious:– Lighting: Lighting: ray sourcesray sources

– Optics:Optics: ray bending/folding devices ray bending/folding devices

– Sensor:Sensor: measure light measure light

– Processing:Processing: assess it assess it

– Display:Display: reproduce it reproduce it

• Ancient Greeks:Ancient Greeks:‘eye rays’ wipe the world‘eye rays’ wipe the worldto feel its contents…to feel its contents…

http://www.mlahanas.de/Greeks/Optics.htmhttp://www.mlahanas.de/Greeks/Optics.htm

Page 13: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

The Photographic Signal PathThe Photographic Signal PathThe Photographic Signal PathThe Photographic Signal Path

Computing can improve every component:Computing can improve every component:

Light SourcesLight Sources SensorsSensorsData Types,Data Types,ProcessingProcessing

DisplayDisplayRaysRays

OpticsOpticsOpticsOptics

““Scene”Scene”RaysRays

Page 14: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Review: How many Rays in a 3-D Scene?Review: How many Rays in a 3-D Scene?Review: How many Rays in a 3-D Scene?Review: How many Rays in a 3-D Scene?

A 4-D set of infinitesimal members. A 4-D set of infinitesimal members.

Imagine:Imagine:

– Convex Enclosure of a 3D scene Convex Enclosure of a 3D scene

– Inward-facing ray camera at every surface pointInward-facing ray camera at every surface point

– Pick the rays you need for ANY camera outside.Pick the rays you need for ANY camera outside.

– 2D surface of cameras,2D surface of cameras,2D 2D ray set for each camera,ray set for each camera,

4D set of rays.4D set of rays.

(Levoy et al. SIGG’96)(Levoy et al. SIGG’96) (Gortler et al. ‘96) (Gortler et al. ‘96)

++

Page 15: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

4-D Light Field / Lumigraph4-D Light Field / Lumigraph4-D Light Field / Lumigraph4-D Light Field / Lumigraph

Measure all the Measure all the outgoingoutgoing light rays. light rays.

Page 16: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

4-D Illumination Field4-D Illumination Field4-D Illumination Field4-D Illumination Field

Same Idea: Measure all the Same Idea: Measure all the incomingincoming light rays light rays

Page 17: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

4D x 4D = 8-D Reflectance Field4D x 4D = 8-D Reflectance Field4D x 4D = 8-D Reflectance Field4D x 4D = 8-D Reflectance Field

Ratio:Ratio: R Rijij = (outgoing ray = (outgoing rayii) / (incoming ray) / (incoming rayjj))

Page 18: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Future Photography:Future Photography:Novel IlluminatorsNovel Illuminators

Novel CamerasNovel Cameras

SceneScene: : 8D Ray Modulator8D Ray Modulator

Generalized Generalized SensorSensor

GeneralizedGeneralizedProcessingProcessing 4D Ray 4D Ray

SamplerSampler

Ray ReconstructorRay Reconstructor

General Optics:General Optics: 4D Ray Benders 4D Ray Benders

Recreated 4D Light fieldRecreated 4D Light field

LightsLights

ModulatorsModulators

4D Incident Lighting4D Incident Lighting

Vie

wed

4D

Lig

ht

Fie

ld

Vie

wed

4D

Lig

ht

Fie

ld

Ge

ner

al O

ptic

s:G

en

eral

Op

tics:

4D

Ray

Ben

der

s4D

Ray

Ben

der

s

Generalized DisplayGeneralized Display

Novel DisplaysNovel Displays

Page 19: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Expand Optics Into SoftwareExpand Optics Into SoftwareExpand Optics Into SoftwareExpand Optics Into Software

Programmable Optical Devices Programmable Optical Devices enable new forms of:enable new forms of:

[ Nayar ][ Nayar ]

• Omni-Directional Lens Systems Omni-Directional Lens Systems

(Hi-Def 360(Hi-Def 360oo video…) video…)

• ‘‘Assorted Pixels’ SensorsAssorted Pixels’ Sensors(Robust HDR, multispectral…)(Robust HDR, multispectral…)

• Lensless Adaptive-Aperture CamerasLensless Adaptive-Aperture Cameras (tracking without panning…)(tracking without panning…)

Page 20: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Expand Scientific / Medical ImagingExpand Scientific / Medical ImagingExpand Scientific / Medical ImagingExpand Scientific / Medical Imaging

4D light sources + 4D cameras 4D light sources + 4D cameras enable new forms of:enable new forms of:

[ Levoy ][ Levoy ]

• Synthetic Aperture Imaging Synthetic Aperture Imaging (see through trees…) (see through trees…)

• Tomography Tomography (3-D volumetric imaging…)(3-D volumetric imaging…)

• Confocal ScanningConfocal Scanning (look inside muddy water…)(look inside muddy water…)

Page 21: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

‘‘The Ideal Photographic Signal’The Ideal Photographic Signal’‘‘The Ideal Photographic Signal’The Ideal Photographic Signal’

I CLAIM IT IS:I CLAIM IT IS:

All Rays? Some Rays? All Rays? Some Rays? ChangesChanges in Some Rays in Some Rays

Photographic ray space is vast and redundantPhotographic ray space is vast and redundant

>8 dimensions: 4D view, 4D light, time, >8 dimensions: 4D view, 4D light, time, ,,

? Gather only ‘? Gather only ‘visually significantvisually significant’ ray changes ?’ ray changes ?

? What rays should we measure ? ? What rays should we measure ?

? How should we combine them ?? How should we combine them ?

? How should we display them ?? How should we display them ?

Page 22: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Beyond ‘Film-Like’ PhotographyBeyond ‘Film-Like’ PhotographyBeyond ‘Film-Like’ PhotographyBeyond ‘Film-Like’ Photography

Call itCall it ‘Computational Photography’: ‘Computational Photography’:

To make ‘meaningful ray changes’To make ‘meaningful ray changes’ tangible, tangible,

• OpticsOptics can do more… can do more…

• Sensors Sensors can do more… can do more…

• Light SourcesLight Sources can do more… can do more…

• ProcessingProcessing can do more… can do more…

by applying low-cost storage, by applying low-cost storage, computation, and control. computation, and control.

Page 23: OUTLINEOUTLINE What is Photography? What is Photography? What is ‘The Photographic Signal’? What is ‘The Photographic Signal’? Perfecting Film-Like Photography:

Course 15:

Computational PhotographyCourse 15:

Computational PhotographyA.2: ConceptsA.2: Concepts Tumblin Tumblin