Download - Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Transcript
Page 1: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Computational Photography

Matthias Zwicker University of Bern

Fall 2012

Page 2: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Today • Course organization

• Course overview

• Image formation

Page 3: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Course organization Instructor

• Matthias Zwicker ([email protected])

Teaching Assistant

• Daniel Donatsch ([email protected])

Page 4: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Course organization Lecture

• Mondays, 14:00-16:00

• Engehaldenstrasse 8, Room 3

Exercises

• Mondays, 16:00-17:00

• Engehaldenstrasse 8, Room 3

Page 5: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Class web page • Class overview

http://www.cgg.unibe.ch/teaching/computational-photography

Page 6: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

ILIAS • Use your campus account to log in

• Join course Magazin → Weitere Institutionen; Weiterbildungen und Studiengänge → BeNeFri Joint Master in Computer Science → HS2012 → 2012HS: 31051 Computational Photography

• Lecture slides

• Exercise description & material

• Additional reading material

• Forum

– Any questions and discussions related to class material and exercises

Page 7: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Exercises • 6 assignments

• Programming projects

– Matlab – Available in ExWi pool

• Exercises on paper

Page 8: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Exercises • Final grade: 40% exercises, 60% final exam • To qualify for final exam: need 70% of

exercise score • Late penalty

– 50% of original score – Exceptions for military service, illness

• Collaboration – Discussion among students is encouraged – Each student must write up and turn in his/her

own solution – If we detect copied material, you will need to talk

to us and explain your material in person; if we are not satisfied, you will not get credit

Page 9: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Final exam • Written, 105 minutes

• Bring two A4 sheets (4 pages) of hand written notes

• Relevant material: slides and exercises

– Wikipedia links not part of class material, but may be useful to better understand concepts discussed in class

• Date: February 2013

Page 10: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Prerequisites • Familiarity with

– Linear algebra (matrix calculations, linear systems of equations, least squares problems)

– Programming experience

Page 11: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Today • Course organization

• Course overview

• Image formation

Page 12: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Computational photography Topics of this class

• Role of computation, algorithms in digital photography today

• Algorithms to extend and improve capabilities of digital photography in the future

Page 14: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Computational photography • More than digital photography

• Arbitrary computation between light measurement and final image – Light measured on sensor is not final image – Computation enhances and extends capabilities of

digital photography • Two „types“ of computation

1. Post-process after traditional imaging 2. Design of new camera devices that require

computation to form an image • Overview of recent research

http://en.wikipedia.org/wiki/Computational_photography

Page 15: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Removing imaging artifacts • Denoising & deblurring

http://www.cs.ust.hk/~quan/publications/yuan-deblur-siggraph07.pdf

Blurry + Output Noisy Algorithm

Page 16: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Removing imaging artifacts • High dynamic range images & tone mapping

Page 17: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Image manipulation • Panoramas

http://en.wikipedia.org/wiki/Image_stitching

Page 18: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Computational optics

Coded aperture

Captured image, slightly blurry everywhere

Page 19: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Computational optics

Recovered depth

Refocused image Sharp foreground, blurry background

Page 20: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Focus of class • „Fun with digital photography and

computer programming“

– Algorithms and computational techniques with potential applications in the consumer domain

– Mostly software, less hardware

• Recent research

Page 21: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

What you will learn • Basic understanding of photography, light,

and color • Practical experience with implementation of

algorithms for image processing & computational photography

• Cool and creative applications of mathematical tools – Fourier transforms – Linear and non linear filtering – Optimization techniques (least squares, iteratively

re-weighted least squares, graph cuts) – Probabilistic models

• Many applications beyond processing images!

Page 22: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Related areas, not covered • Image processing for scientific applications

– Physics, biology, etc.

• Optics, lens design

• Photosensors, sensor design

• Computational imaging

– Tomography, radar, microscopy

• 3D imaging

• Using photo processing tools, e.g. Photoshop

• Artistical aspects of photography

Page 23: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Syllabus 1. Introduction, image formation 2. Color & color processing 3. Dynamic range & contrast 4. Sampling, reconstruction, & the frequency domain 5. Image restoration: denoising & deblurring 6. Image manipulation using optimization 7. Gradient domain image manipulation 8. Warping & morphing 9. Panoramas 10. Automatic alignment 11. Probabilistic image models 12. Light fields 13. Capturing light transport

http://www.cgg.unibe.ch/teaching/computational-photography

Page 24: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

• Cameras, image artifacts

Image formation

Page 25: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Color • Color perception, color spaces, color

measurement, color processing

Page 26: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Dynamic range & contrast • HDR imaging

http://en.wikipedia.org/wiki/High_dynamic_range_imaging http://en.wikipedia.org/wiki/Tone_mapping

Page 27: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Sampling, reconstruction • Sampling artifacts

• Frequency domain analysis

Spatial Domain Frequency Domain

Page 28: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Image restoration • Denoising & deblurring

Blurry input Deblurred output

Estimated blur kernel (scaled) http://vision.ucsd.edu/kriegman-grp/research/psf_estimation/

Page 29: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Image manipulation using optimization • Photomontage, matting, colorization

http://grail.cs.washington.edu/projects/photomontage/

http://www.cs.huji.ac.il/~yweiss/Colorization/

http://grail.cs.washington.edu/projects/digital-matting/image-matting/

Page 30: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Gradient domain manipulation • Poisson equation

http://portal.acm.org/citation.cfm?id=882269

Page 31: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Warping & morphing

Page 33: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Probabilistic models • Faces, textures

http://web4.cs.ucl.ac.uk/staff/j.kautz/publications/Visio_SIG09.pdf

Page 36: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Today • Course organization

• Course overview

• Image formation

Page 37: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Models of light

Page 38: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Question • Why is there no image on a white piece of

paper?

Page 39: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Question • Why is there no image on a white piece of

paper?

• Receives all light rays

– Images from all viewpoints

• Need to select light rays for specifice image, viewpoint

• How?

Page 40: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

• Invented by Alhazen, 10th century http://en.wikipedia.org/wiki/Pinhole_camera

Pinhole camera

Page 41: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Limitations • Small pinhole: sharper image, longer exposure

• Larger pinhole: blurrier image, shorter exposure

Page 42: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Camera model • Thin lens, aperture, shutter, film

Page 43: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Lenses • Gather more light

– Proportional to area of lens aperture

• Sharp image if focused

– Use refraction http://en.wikipedia.org/wiki/Refraction

Lens

http://en.wikipedia.org/wiki/Lens_(optics)

Scene point (emits or

reflects light) Image of

scene point

Page 44: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Lenses Pinhole Lens

6 sec. exposure 0.01 sec exposure

Page 45: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Thin lens model http://en.wikipedia.org/wiki/Thin_lens

• Theoretical model for well-behaved lenses

• Properties

1. All parallel rays converge at focal length

2. Rays through the center are not deflected

Same perspective image as pinhole placed at center of lens

Page 46: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Thin lens model • How are arbitrary rays deflected when

passing through a thin lens?

1. Parallel rays converge at focal length f

Page 47: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Thin lens model 2. Rays through center are not deflected

Page 48: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Thin lens model • Similar triangles

Page 49: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Thin lens model • More similar triangles

Page 50: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Thin lens model • Thin lens formula

• All rays passing through a single point y on a plane at distance in front of the lens will pass through a single point y’ at distance behind the lens

Page 51: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Thin lens model • Focus at infinity:

– Film at distance f

• Closest focusing distance:

– Film at infinity

Film plane

Object

Page 52: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Thin lens model • Out of focus film plane results in spherical

blur

Out of focus film planes

Spherical blur

Page 53: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Properties of real lenses • Mostly undesired!

• Aberrations

– Spherical aberration – Chromatic aberration

• Distortion

– Barrel distortion – Pincushion distortion

• Etc.

Barrel & pincushion distortion

Page 54: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Question • What‘s the advantage of a lens with a

short focal length? In what situation would this be useful?

• What‘s the advantage of a lens with a long focal length? In what situation would this be useful?

Page 55: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Camera model • Thin lens, aperture, shutter, film

Page 56: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Aperture • Amount of light captured at sensor

proportional to area of aperture

Aperture

Page 57: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Aperture • Blurriness of out of focus objects depends

on aperture size

• Aperture size determines depth of field: depth range that is sharp in image

Aperture

Page 58: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Depth of field

Page 59: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Circle of confusion • Also called „blur circle“

• Calculation of radius c – Lens focused at S1

– Object at S2 – Aperture A – Focal length f sensor

http://en.wikipedia.org/wiki/Circle_of_confusion

Proportional to A

Page 60: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

f-number http://en.wikipedia.org/wiki/F-number

• Definition: (focal length) / (diameter of aperture)

• Large aperture means small f-number

• Practice: f-stops increase by factors of

– f/2.0, f/2.8, f/4, f/5.6, f/8 – Aperture area gets halved in each step

Page 61: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

f-number • Smaller f-number

– Larger aperture – Capture more light – Small (shallow) depth of field

• Larger f-number

– Smaller aperture – Capture less light – Large depth of field

Page 62: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Camera model • Thin lens, aperture, shutter, film

Page 63: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Shutter speed http://en.wikipedia.org/wiki/Shutter_speed

• Determines time the film is exposed to light

• Amount of light captured is proportional to exposure time

• Long exposure leads to motion blur

Page 64: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Reciprocity • Amount of light captured stays

same if exposure is doubled and aperture area is halved (or vice versa)

Page 65: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Reciprocity • Which exposure/aperture combination?

Page 66: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Camera model • Thin lens, aperture, shutter, film

Page 67: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Film • Film/sensor responds roughly linearly to light

– „Double the amount of light leads to double the recorded value“

• Film speed: sensitivity of film to light

– Digital photography analog: sensor gain (scaling or amplification factor)

• Measured using ISO scale

– Linear: sensitivity is proportional to ISO value – „Double ISO value, halve the exposure time, get

same recorded value“

Page 68: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Film • Trade-off: higher gain, more noise

ISO 100 ISO 3200

Page 69: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Film • Underexposure

– Not enough light, image too dark • Overexposure

– Film or sensor is saturated – Clipping of highlight details

Good exposure Overexposure Underexposure

Page 70: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Conclusions • Simple camera model

– Thin lens, aperture, shutter, film

• Photographs often have undesired artifacts

– Distortions, color artifacts, blur, noise, under/overexposure

Goal

• Develop algorithms to remove artifacts after image is captured

Page 71: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

References • „Photography“, by London, Upton, Stone

Page 72: Computational Photography - Portal · Computational photography • More than digital photography • Arbitrary computation between light measurement and final image – Light measured

Next time • Color, color processing