Computational photography

Kazuhiro Sato

Computer Vision: Algorithms and Applications

Chapter 10

What is Computational Photography?

To enhance or extend the capabilities of digital photography

enhanceextend

Contents –computational photography-

1. Photometric calibration2. High dynamic range imaging3. Super-resolution and blur

removal4. Image matting and

compositing5. Texture analysis and

synthesis

1. Photometric calibration

By the way,

How does an image

come out?

imaging pipeline

First, we characterize• the mapping

functions• the amounts of noise

The mapping functions consists of

• the radiometric response function

• vignetting• the point spread function

1.1 Radiometric Response Function

fin

al bri

ghtn

ess

scene radiane

optics𝑀

𝐼sensor plane

often NOT linear

How can we determine the function?

• calibration chart

• polynomial approximation 𝐼=∑

𝑛=0

𝑁

𝑐𝑛𝑀𝑛

• least squares (explained later)

1.2 Noise Level Estimation

1. Segment the input image

2. Fit a linear function

3. Plot the standard deviation

4. Fit a lower envelope

estimated noise level

pixel value

𝜎

[Liu et al. ’08]

1.3 Vignetting

・ Radial Gradient Symmetry [Zheng et al. ’08]

∆ 𝐼 (𝑥 , 𝑦)

𝜕 𝐼𝜕𝑟

0− +¿𝜕 𝐼𝜕𝑟

Histogram (asymmetry)

𝑍 (𝑟 , 𝜃 )=𝐼 (𝑟 ,𝜃 )𝑉 (𝑟 )

Find the optimal that minimizes the asymmetry using “maximum a posteriori (MAP)”

: the image with vignetting: the vignetting-free image: the vignetting function

𝑉=argmax𝑉

𝑃 (𝑉∨𝑍 )∝argmax𝑉

𝑃 (𝑍∨𝑉 ) 𝑃 (𝑉 )

1.4 Optical Blur Estimation

・ PSF Estimation [Joshi et al. ’08]

Estimating the PSF kernel using a calibration pattern𝐾=argmin

𝐾‖𝐵−𝐷 ( 𝐼 ∗ 𝐾 )‖2

: sensed (blurred) image: predicted (sharp) image: downsampling operator

Solving a Bayesian framework

using a maximum a posteriori (MAP)

argmax𝐾

𝑃 (𝐾∨𝐵 )=argmax𝐾

𝑃 (𝐵∨𝐾 ) 𝑃 (𝐾 )𝑃 (𝐵 )

𝐿 (𝐵∨𝐾 )=‖𝑀 (𝐵 )−𝑀 ( 𝐼⊗ 𝐾 )‖2

𝜎 2

¿ argmin𝐾𝐿 (𝐵∨𝐾 )+𝐿 (𝐾 )

𝐿 (𝐾 )= λ𝛾‖𝛻𝐾‖2

・ Recovering the PSF without calibration

1. Fit step edges to the elongated ones

2. Apply the least squares for every surrounded pixel𝐾=argmin

𝐾‖𝐵−𝐷 ( 𝐼 ∗ 𝐾 )‖2

2. High Dynamic Range Imaging

Dynamic Range

0.01 1 100 10000

moonlight sunlight

Luminance[cd/m2]

indoor lighting

Human eye

CCD camera

The natural world is too bright to be captured!

Creating a properly exposed photo (High dynamic range imaging)

different exposures

properly exposed photo

create

How to create such a photo?

1. Estimate the radiometric response function

2. Estimate a radiance map

3. Tone map the resulting HDR image into a 8-bit one

1. Estimate the radiometric response function

𝑧𝑖𝑗= 𝑓 (𝐸𝑖 𝑡 𝑗 )log 𝐸 𝑖

𝑧 𝑖𝑗

𝑓 −1 (𝑧 𝑖𝑗 )=𝐸𝑖 𝑡 𝑗

th exposure ( )

𝑔 ( 𝑧𝑖𝑗 )=log 𝑓 −1 (𝑧𝑖𝑗 )=log 𝐸𝑖+ log 𝑡 𝑗

[Debevec et al. ’97]

min𝐸=∑𝑖∑𝑗

𝑤 (𝑧𝑖𝑗 ) [𝑔 (𝑧𝑖𝑗 )−log 𝐸 𝑖− log 𝑡 𝑗 ]2+¿ λ∑𝑘

𝑤 (𝑘 )𝑔′ ′ (𝑘 )2 ¿

log 𝐸 𝑖

𝑧 𝑖𝑗

h𝑤 𝑒𝑟𝑒𝑤 (𝑧 )={𝑧−𝑧𝑚𝑖𝑛 𝑧≤ (𝑧𝑚𝑖𝑛+𝑧𝑚𝑎𝑥) /2𝑧𝑚𝑎𝑥− 𝑧 𝑧>( 𝑧𝑚𝑖𝑛+ 𝑧𝑚𝑎𝑥 )/2

th exposure ( )

2. Estimate a radiance map[Mitsunaga et al. ’99]

different exposures

𝑧𝑖 ,1 𝑧𝑖 ,2

𝑧𝑖 ,1

𝑧𝑖 ,2

Merging the input images into a composite radiance map.

log 𝐸 𝑖=∑𝑗

𝑤 ( 𝑧𝑖𝑗 ) [𝑔 (𝑧 𝑖𝑗 )− log 𝑡 𝑗 ]∑𝑗

𝑤 (𝑧𝑖𝑗 )

h𝑤 𝑒𝑟𝑒𝑤 (𝑧 )=𝑔 (𝑧 )/𝑔′ (𝑧 )

different exposures

𝑧𝑖 ,1 𝑧𝑖 ,2

radiance map (grayscale)

merge

3. Tone map the resulting HDR image into a 8-bit one

bits / pixel

8-bit image

HDR image

8 bits / pixel

input HDR image gamma applied to each channel

gamma applied to luminance only

This global approach fails to preserve details in regions with widely varying exposures.

・ Global tone mapping using a transfer curve[Larson et al. ’97]

2.1 Tone mapping

・ Local tone mapping using bilateral filter[Durand et al. ’02]

This approach doesn’t create visible halos around the edges.

result withlow-pass filtering

(visible halos)

result withbilateral filtering

(no halos)

・ Gradient domain tone mapping [Fattal et al. ’02]

𝐿𝑖𝑛

𝐿𝑜𝑢𝑡

𝐶𝑜𝑢𝑡=(𝐶𝑖𝑛

𝐿𝑖𝑛)𝑠

𝐿𝑜𝑢𝑡𝑠∈ [0.4 ,0.6 ]

The new luminance is combined with the original color image.

𝐺 (𝑥 )=𝐻 ′ (𝑥 )Φ (𝑥 )

Attenuation map Tone-mapped result

2.2 Flash photography [Petschnigg et al. 04]

Combine flash and non-flash images to achieve better photos

Joint bilateral filter’s kernel

domain kernel

𝑤 (𝑖 , 𝑗 ,𝑘 , 𝑙 )=exp (− (𝑖−𝑘 )2+( 𝑗−𝑙 )2

2𝜎𝑑2 ) ∙exp(−‖𝐹 (𝑖 , 𝑗 )−𝐹 (𝑘 , 𝑙 )‖2

2𝜎𝑟2 )

range kernel

3. Super-resolution and blur removal

3.1 Color image demosaicing

Bayer RGB patternin a camera sensor

Full-color RGB

interpolate

・ Bayesian demosaicing with a two-color prior

Original Bilinear [Bennett et al. ’06]

[Bennett et al. ’06]

Two-color model [Bennett et al. ’06]

4. Image matting and compositing

Image “matting” and “compositing”?

Input OutputAlpha-matting

matting compositing

What is the problem in matting and compositing?

foreground

background

boundary

𝛼=1𝛼=0

0<𝛼<1

We need the opacity !

Failed example of matting

Alpha matte Composite Inset

Input

4.1 Blue screen matting [Smith et al. ‘96]

𝑐𝑜= [𝑅𝑜𝐺𝑜0 ]𝑐 𝑓=[𝑅 𝑓𝐺 𝑓 𝐵 𝑓 ] 𝑐𝑘=[ 00𝐵𝑘 ]

{ 𝑅𝑜=𝑅 𝑓

𝐺𝑜=𝐺 𝑓

(1−𝛼𝑜 )𝐵𝑘=𝐵 𝑓

solve

)

∴𝐶𝑜=[𝑅𝑜𝐺𝑜0𝛼𝑜 ]

¿ [𝑅𝑓 𝐺 𝑓 01−𝐵 𝑓

𝐵𝑘]

𝑐𝑜= [𝑅𝑜𝐺𝑜𝐵𝑜 ]𝑐 𝑓 1=[𝑅 𝑓 1

𝐺 𝑓 1𝐵 𝑓 1 ]

𝑐𝑘1=[00𝐵𝑘1 ]

solve

)

∴𝐶𝑜=[𝑅𝑜𝐺𝑜𝐵𝑜𝛼𝑜 ]

¿ [𝑅𝑓 1𝐺 𝑓 1

𝐵 𝑓 2𝐵𝑘1−𝐵 𝑓 1

𝐵𝑘2

𝐵𝑘1−𝐵𝑘2

1−𝐵 𝑓 1−𝐵 𝑓 2

𝐵𝑘1−𝐵𝑘2

]

・ Two-screen matting[Smith et al. ’96]

𝑐 𝑓 2=[𝑅 𝑓 2

𝐺 𝑓 2𝐵 𝑓 2 ]

𝑐𝑘2= [00𝐵𝑘2 ]

{𝐵𝑜+ (1−𝛼𝑜 )𝐵𝑘1=𝐵 𝑓 1

𝐵𝑜+(1−𝛼𝑜 )𝐵𝑘2=𝐵 𝑓 2

4.2 Natural image matting

Input Hand-drawn trimap

Alpha map

foreground composite

[Chuang et al. ’01]

・ Knockout[Berman et al. ’00]

𝐹 𝑖

𝐵𝑖

𝐶=𝛼𝐹+ (1−𝛼 ) 𝐵

1. Estimate and

𝐹=∑𝑖

𝑤 𝑖𝐹 𝑖 𝐵 ′=∑𝑖

𝑤 𝑖𝐵𝑖

2. Refine → 𝐹𝐵 ′⊥𝐵𝐵 ′ //

3. Estimate by axis-projection

𝛼𝑅=𝑓 𝑅 (𝐶 )− 𝑓 𝑅 (𝐵 )𝑓 𝑅 (𝐹 )− 𝑓 𝑅 (𝐵 )

,𝛼𝐺 ,𝛼𝐵

𝛼 𝑓𝑖𝑛𝑎𝑙= ∑𝑋=𝑅 ,𝐺 ,𝐵

[ 𝑓 𝑋 (𝐹 )− 𝑓 𝑋 (𝐵 ) ] ∙𝛼 𝑋

・ Bayesian approach[Chuang et al. ’01]

𝐹

𝐵

𝐶observation

most likely estimates

arg max𝐹 ,𝐵 ,𝛼

𝑃 (𝐹 ,𝐵 ,𝛼∨𝐶 )

𝐿 (𝐶|𝐹 ,𝐵 ,𝛼 )=−‖𝐶−𝛼𝐹− (1−𝛼 ) 𝐵‖2

𝜎𝐶2

𝐿 (𝐹 )=− (𝐹−𝐹 )𝑇 Σ𝐹−1 (𝐹−𝐹 )/2

: same as the above except : const.

・ Comparison of natural image mattings

4.3 Optimization-based matting

・ Border matting

1. Get a trimap by hard segmentation

𝑇 𝐹 (𝛼=1 ) ,𝑇 𝑢 ,𝑇 𝐵 (𝛼=0 )

[Rother et al. ’04]

2. Minimizing the energy of mapping

𝐸= ∑𝑛∈𝑇 𝑈

~𝐷𝑛 (𝛼𝑛)+∑𝑡=1

𝑇 ~𝑉 (∆𝑡 ,𝜎𝑡 ,∆𝑡+1 ,𝜎 𝑡+!)

data term smoothness term

・ Color line (closed-form) matting

1. Estimate for each

𝛼 𝑖𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑=𝒂𝒌 ∙ (𝑪 𝒊−𝑩𝟎)=𝒂𝒌 ∙𝑪+𝑏𝑘

2. Minimize the deviations of the actual from the above 𝐸𝛼=∑

𝑘 ( ∑𝑖∈𝑊 𝑘

(𝛼𝑖−𝛼 𝑖𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 )2+𝜖‖𝒂𝒌‖)

data term regularization term

local window local values

[Levin et al. ’08]

3. Eliminate to get a final energy

4.4 Smoke, shadow, and flash matting

・ Smoke matting[Chuang et al. ’02]

input frameremoving the foreground

object

estimated alpha matte

insertion of new objects

・ Shadow matting[Chuang et al. ’03]

4.5 Video matting

[Chuang et al. ’02]

5. Texture analysis and synthesis

Texture synthesis is …

Small patch

Similar-looking larger patch

synthesize

Texture synthesis using non-parametric sampling

Texture synthesis using non-parametric sampling

[Efros et al. ’99]

[Efros et al. ’01]

5.1 hole filling and inpainting

inpaint

Original Inpainted result

・ Exemplar-based inpainting[Criminisi et al. ’04]

1. Compute patch priorities

𝑃 (𝒑 )=𝐶 (𝒑 )𝐷 (𝒑 )

C (𝒑 )=∑

𝒒∈Ψ 𝒑∩ ( 𝑰−𝛀 )

𝐶 (𝒒)

|Ψ 𝒑|

𝐷 (𝒑 )=|𝛻 𝐼𝑝

⊥∙𝒏𝒑|𝛼

data termconfidence term

2. Find the most similar patch to with top priorityΨ �̂�=arg min

Ψ 𝒒∈Φ𝑑 (Ψ �̂� ,Ψ 𝒒 )

3. Update confidence values𝐶 (𝒑 )=𝐶 (�̂� )∀ 𝒑∈Ψ �̂�∩Ω

onio

n p

eel

[Cri

min

isi et

al.

’04

]

original removed region

Remain the gradient alongthe region boundary

5.2 Non-photorealistic rendering

Non-photorealistic rendering using texture synthesis

・ Texture transfer・ Image analogies

・ Texture transfer

[Efros et al. ’01]

input outputtexture transfer

input outputtexture transfer

1. Find the most similar block 𝐸𝑖=𝛼𝐸𝑜𝑣𝑒𝑟𝑙𝑎𝑝+(1−𝛼 ) 𝐸 𝑙𝑢𝑚𝑖𝑛𝑎𝑛𝑐𝑒

𝐸 𝑗

𝐸𝑖

𝐵 𝑗

𝐵𝑖

( means ordinary texture synthesis. )

2. Find the minimum error boundary cut𝑒=(𝐵1

𝑜𝑣  −𝐵2𝑜𝑣   )2

𝐵1𝑜𝑣 ,𝐵2

𝑜𝑣

( the minimal cost path corresponds to the valley in the surface . )

・ Image analogies

[Hertzmann et al. ’01]

𝐴 𝐴 ′ 𝐵 𝐵 ′

?: :¿: :¿

synthesize this image

𝐴 𝐴 ′

𝐵 𝐵 ′

unfiltered example NPR-filtered example

target result

𝐴 𝐴 ′

𝐵 𝐵 ′

unfiltered example filtered example

target result

Thank you!