Chap. 10 computational photography

72
Computational photography Kazuhiro Sato Computer Vision: Algorithms and Applications Chapter 10

description

This is a ppt file for study meetings held in our lab, describing chapter 10 computational photography in the book of Szeliski's "Computer Vision: Algorithms and Applications."Aizawa-Yamasaki Lab. at The Univ. of Tokyo http://www.hal.t.u-tokyo.ac.jp/

Transcript of Chap. 10 computational photography

Page 1: Chap. 10 computational photography

Computational photography

Kazuhiro Sato

Computer Vision: Algorithms and Applications

Chapter 10

Page 2: Chap. 10 computational photography

What is Computational Photography?

Page 3: Chap. 10 computational photography

To enhance or extend the capabilities of digital photography

enhanceextend

Page 4: Chap. 10 computational photography

Contents –computational photography-

1. Photometric calibration2. High dynamic range imaging3. Super-resolution and blur

removal4. Image matting and

compositing5. Texture analysis and

synthesis

Page 5: Chap. 10 computational photography

1. Photometric calibration

Page 6: Chap. 10 computational photography

By the way,

How does an image

come out?

Page 7: Chap. 10 computational photography

imaging pipeline

Page 8: Chap. 10 computational photography

First, we characterizeβ€’ the mapping

functionsβ€’ the amounts of noise

Page 9: Chap. 10 computational photography

The mapping functions consists of

β€’ the radiometric response function

β€’ vignettingβ€’ the point spread function

Page 10: Chap. 10 computational photography

1.1 Radiometric Response Function

fin

al bri

ghtn

ess

scene radiane

optics𝑀

𝐼sensor plane

often NOT linear

Page 11: Chap. 10 computational photography

How can we determine the function?

β€’ calibration chart

β€’ polynomial approximation 𝐼=βˆ‘

𝑛=0

𝑁

𝑐𝑛𝑀𝑛

β€’ least squares (explained later)

Page 12: Chap. 10 computational photography

1.2 Noise Level Estimation

1. Segment the input image

2. Fit a linear function

3. Plot the standard deviation

4. Fit a lower envelope

estimated noise level

pixel value

𝜎

[Liu et al. ’08]

Page 13: Chap. 10 computational photography

1.3 Vignetting

・ Radial Gradient Symmetry [Zheng et al. ’08]

βˆ† 𝐼 (π‘₯ , 𝑦)

πœ• πΌπœ•π‘Ÿ

0βˆ’ +ΒΏπœ• πΌπœ•π‘Ÿ

Histogram (asymmetry)

Page 14: Chap. 10 computational photography

𝑍 (π‘Ÿ , πœƒ )=𝐼 (π‘Ÿ ,πœƒ )𝑉 (π‘Ÿ )

Find the optimal that minimizes the asymmetry using β€œmaximum a posteriori (MAP)”

: the image with vignetting: the vignetting-free image: the vignetting function

𝑉=argmax𝑉

𝑃 (π‘‰βˆ¨π‘ )∝argmax𝑉

𝑃 (π‘βˆ¨π‘‰ ) 𝑃 (𝑉 )

Page 15: Chap. 10 computational photography
Page 16: Chap. 10 computational photography

1.4 Optical Blur Estimation

・ PSF Estimation [Joshi et al. ’08]

Estimating the PSF kernel using a calibration pattern𝐾=argmin

πΎβ€–π΅βˆ’π· ( 𝐼 βˆ— 𝐾 )β€–2

: sensed (blurred) image: predicted (sharp) image: downsampling operator

Page 17: Chap. 10 computational photography

Solving a Bayesian framework

using a maximum a posteriori (MAP)

argmax𝐾

𝑃 (𝐾∨𝐡 )=argmax𝐾

𝑃 (𝐡∨𝐾 ) 𝑃 (𝐾 )𝑃 (𝐡 )

𝐿 (𝐡∨𝐾 )=‖𝑀 (𝐡 )βˆ’π‘€ ( πΌβŠ— 𝐾 )β€–2

𝜎 2

¿ argmin𝐾𝐿 (𝐡∨𝐾 )+𝐿 (𝐾 )

𝐿 (𝐾 )= λ𝛾‖𝛻𝐾‖2

Page 18: Chap. 10 computational photography

・ Recovering the PSF without calibration

1. Fit step edges to the elongated ones

2. Apply the least squares for every surrounded pixel𝐾=argmin

πΎβ€–π΅βˆ’π· ( 𝐼 βˆ— 𝐾 )β€–2

Page 19: Chap. 10 computational photography

2. High Dynamic Range Imaging

Page 20: Chap. 10 computational photography

Dynamic Range

0.01 1 100 10000

moonlight sunlight

Luminance[cd/m2]

indoor lighting

Human eye

CCD camera

Page 21: Chap. 10 computational photography

The natural world is too bright to be captured!

Page 22: Chap. 10 computational photography

Creating a properly exposed photo (High dynamic range imaging)

different exposures

properly exposed photo

create

Page 23: Chap. 10 computational photography

How to create such a photo?

1. Estimate the radiometric response function

2. Estimate a radiance map

3. Tone map the resulting HDR image into a 8-bit one

Page 24: Chap. 10 computational photography

1. Estimate the radiometric response function

𝑧𝑖𝑗= 𝑓 (𝐸𝑖 𝑑 𝑗 )log 𝐸 𝑖

𝑧 𝑖𝑗

𝑓 βˆ’1 (𝑧 𝑖𝑗 )=𝐸𝑖 𝑑 𝑗

th exposure ( )

𝑔 ( 𝑧𝑖𝑗 )=log 𝑓 βˆ’1 (𝑧𝑖𝑗 )=log 𝐸𝑖+ log 𝑑 𝑗

[Debevec et al. ’97]

Page 25: Chap. 10 computational photography

min𝐸=βˆ‘π‘–βˆ‘π‘—

𝑀 (𝑧𝑖𝑗 ) [𝑔 (𝑧𝑖𝑗 )βˆ’log 𝐸 π‘–βˆ’ log 𝑑 𝑗 ]2+ΒΏ Ξ»βˆ‘π‘˜

𝑀 (π‘˜ )𝑔′ β€² (π‘˜ )2 ΒΏ

log 𝐸 𝑖

𝑧 𝑖𝑗

h𝑀 π‘’π‘Ÿπ‘’π‘€ (𝑧 )={π‘§βˆ’π‘§π‘šπ‘–π‘› 𝑧≀ (π‘§π‘šπ‘–π‘›+π‘§π‘šπ‘Žπ‘₯) /2π‘§π‘šπ‘Žπ‘₯βˆ’ 𝑧 𝑧>( π‘§π‘šπ‘–π‘›+ π‘§π‘šπ‘Žπ‘₯ )/2

th exposure ( )

Page 26: Chap. 10 computational photography

2. Estimate a radiance map[Mitsunaga et al. ’99]

different exposures

𝑧𝑖 ,1 𝑧𝑖 ,2

𝑧𝑖 ,1

𝑧𝑖 ,2

Merging the input images into a composite radiance map.

Page 27: Chap. 10 computational photography

log 𝐸 𝑖=βˆ‘π‘—

𝑀 ( 𝑧𝑖𝑗 ) [𝑔 (𝑧 𝑖𝑗 )βˆ’ log 𝑑 𝑗 ]βˆ‘π‘—

𝑀 (𝑧𝑖𝑗 )

h𝑀 π‘’π‘Ÿπ‘’π‘€ (𝑧 )=𝑔 (𝑧 )/𝑔′ (𝑧 )

different exposures

𝑧𝑖 ,1 𝑧𝑖 ,2

radiance map (grayscale)

merge

Page 28: Chap. 10 computational photography

3. Tone map the resulting HDR image into a 8-bit one

bits / pixel

8-bit image

HDR image

8 bits / pixel

Page 29: Chap. 10 computational photography

input HDR image gamma applied to each channel

gamma applied to luminance only

This global approach fails to preserve details in regions with widely varying exposures.

・ Global tone mapping using a transfer curve[Larson et al. ’97]

2.1 Tone mapping

Page 30: Chap. 10 computational photography

・ Local tone mapping using bilateral filter[Durand et al. ’02]

Page 31: Chap. 10 computational photography

This approach doesn’t create visible halos around the edges.

result withlow-pass filtering

(visible halos)

result withbilateral filtering

(no halos)

Page 32: Chap. 10 computational photography

・ Gradient domain tone mapping [Fattal et al. ’02]

𝐿𝑖𝑛

πΏπ‘œπ‘’π‘‘

πΆπ‘œπ‘’π‘‘=(𝐢𝑖𝑛

𝐿𝑖𝑛)𝑠

πΏπ‘œπ‘’π‘‘π‘ βˆˆ [0.4 ,0.6 ]

The new luminance is combined with the original color image.

𝐺 (π‘₯ )=𝐻 β€² (π‘₯ )Ξ¦ (π‘₯ )

Page 33: Chap. 10 computational photography

Attenuation map Tone-mapped result

Page 34: Chap. 10 computational photography

2.2 Flash photography [Petschnigg et al. 04]

Combine flash and non-flash images to achieve better photos

Page 35: Chap. 10 computational photography

Joint bilateral filter’s kernel

domain kernel

𝑀 (𝑖 , 𝑗 ,π‘˜ , 𝑙 )=exp (βˆ’ (π‘–βˆ’π‘˜ )2+( π‘—βˆ’π‘™ )2

2πœŽπ‘‘2 ) βˆ™exp(βˆ’β€–πΉ (𝑖 , 𝑗 )βˆ’πΉ (π‘˜ , 𝑙 )β€–2

2πœŽπ‘Ÿ2 )

range kernel

Page 36: Chap. 10 computational photography

3. Super-resolution and blur removal

Page 37: Chap. 10 computational photography

3.1 Color image demosaicing

Bayer RGB patternin a camera sensor

Full-color RGB

interpolate

Page 38: Chap. 10 computational photography

・ Bayesian demosaicing with a two-color prior

Original Bilinear [Bennett et al. ’06]

[Bennett et al. ’06]

Page 39: Chap. 10 computational photography

Two-color model [Bennett et al. ’06]

Page 40: Chap. 10 computational photography

4. Image matting and compositing

Page 41: Chap. 10 computational photography

Image β€œmatting” and β€œcompositing”?

Input OutputAlpha-matting

matting compositing

Page 42: Chap. 10 computational photography

What is the problem in matting and compositing?

foreground

background

boundary

𝛼=1𝛼=0

0<𝛼<1

We need the opacity !

Page 43: Chap. 10 computational photography

Failed example of matting

Alpha matte Composite Inset

Input

Page 44: Chap. 10 computational photography

4.1 Blue screen matting [Smith et al. β€˜96]

π‘π‘œ= [π‘…π‘œπΊπ‘œ0 ]𝑐 𝑓=[𝑅 𝑓𝐺 𝑓 𝐡 𝑓 ] π‘π‘˜=[ 00π΅π‘˜ ]

{ π‘…π‘œ=𝑅 𝑓

πΊπ‘œ=𝐺 𝑓

(1βˆ’π›Όπ‘œ )π΅π‘˜=𝐡 𝑓

solve

)

βˆ΄πΆπ‘œ=[π‘…π‘œπΊπ‘œ0π›Όπ‘œ ]

ΒΏ [𝑅𝑓 𝐺 𝑓 01βˆ’π΅ 𝑓

π΅π‘˜]

Page 45: Chap. 10 computational photography

π‘π‘œ= [π‘…π‘œπΊπ‘œπ΅π‘œ ]𝑐 𝑓 1=[𝑅 𝑓 1

𝐺 𝑓 1𝐡 𝑓 1 ]

π‘π‘˜1=[00π΅π‘˜1 ]

solve

)

βˆ΄πΆπ‘œ=[π‘…π‘œπΊπ‘œπ΅π‘œπ›Όπ‘œ ]

ΒΏ [𝑅𝑓 1𝐺 𝑓 1

𝐡 𝑓 2π΅π‘˜1βˆ’π΅ 𝑓 1

π΅π‘˜2

π΅π‘˜1βˆ’π΅π‘˜2

1βˆ’π΅ 𝑓 1βˆ’π΅ 𝑓 2

π΅π‘˜1βˆ’π΅π‘˜2

]

・ Two-screen matting[Smith et al. ’96]

𝑐 𝑓 2=[𝑅 𝑓 2

𝐺 𝑓 2𝐡 𝑓 2 ]

π‘π‘˜2= [00π΅π‘˜2 ]

{π΅π‘œ+ (1βˆ’π›Όπ‘œ )π΅π‘˜1=𝐡 𝑓 1

π΅π‘œ+(1βˆ’π›Όπ‘œ )π΅π‘˜2=𝐡 𝑓 2

Page 46: Chap. 10 computational photography

4.2 Natural image matting

Input Hand-drawn trimap

Alpha map

foreground composite

[Chuang et al. ’01]

Page 47: Chap. 10 computational photography

・ Knockout[Berman et al. ’00]

𝐹 𝑖

𝐡𝑖

𝐢=𝛼𝐹+ (1βˆ’π›Ό ) 𝐡

1. Estimate and

𝐹=βˆ‘π‘–

𝑀 𝑖𝐹 𝑖 𝐡 β€²=βˆ‘π‘–

𝑀 𝑖𝐡𝑖

2. Refine β†’ 𝐹𝐡 β€²βŠ₯𝐡𝐡 β€² //

3. Estimate by axis-projection

𝛼𝑅=𝑓 𝑅 (𝐢 )βˆ’ 𝑓 𝑅 (𝐡 )𝑓 𝑅 (𝐹 )βˆ’ 𝑓 𝑅 (𝐡 )

,𝛼𝐺 ,𝛼𝐡

𝛼 π‘“π‘–π‘›π‘Žπ‘™= βˆ‘π‘‹=𝑅 ,𝐺 ,𝐡

[ 𝑓 𝑋 (𝐹 )βˆ’ 𝑓 𝑋 (𝐡 ) ] βˆ™π›Ό 𝑋

Page 48: Chap. 10 computational photography

・ Bayesian approach[Chuang et al. ’01]

𝐹

𝐡

𝐢observation

most likely estimates

arg max𝐹 ,𝐡 ,𝛼

𝑃 (𝐹 ,𝐡 ,π›Όβˆ¨πΆ )

Page 49: Chap. 10 computational photography

𝐿 (𝐢|𝐹 ,𝐡 ,𝛼 )=βˆ’β€–πΆβˆ’π›ΌπΉβˆ’ (1βˆ’π›Ό ) 𝐡‖2

𝜎𝐢2

𝐿 (𝐹 )=βˆ’ (πΉβˆ’πΉ )𝑇 Ξ£πΉβˆ’1 (πΉβˆ’πΉ )/2

: same as the above except : const.

Page 50: Chap. 10 computational photography

・ Comparison of natural image mattings

Page 51: Chap. 10 computational photography

4.3 Optimization-based matting

・ Border matting

1. Get a trimap by hard segmentation

𝑇 𝐹 (𝛼=1 ) ,𝑇 𝑒 ,𝑇 𝐡 (𝛼=0 )

[Rother et al. ’04]

Page 52: Chap. 10 computational photography

2. Minimizing the energy of mapping

𝐸= βˆ‘π‘›βˆˆπ‘‡ π‘ˆ

~𝐷𝑛 (𝛼𝑛)+βˆ‘π‘‘=1

𝑇 ~𝑉 (βˆ†π‘‘ ,πœŽπ‘‘ ,βˆ†π‘‘+1 ,𝜎 𝑑+!)

data term smoothness term

Page 53: Chap. 10 computational photography

・ Color line (closed-form) matting

1. Estimate for each

𝛼 π‘–π‘’π‘ π‘‘π‘–π‘šπ‘Žπ‘‘π‘’π‘‘=π’‚π’Œ βˆ™ (π‘ͺ π’Šβˆ’π‘©πŸŽ)=π’‚π’Œ βˆ™π‘ͺ+π‘π‘˜

2. Minimize the deviations of the actual from the above 𝐸𝛼=βˆ‘

π‘˜ ( βˆ‘π‘–βˆˆπ‘Š π‘˜

(π›Όπ‘–βˆ’π›Ό π‘–π‘’π‘ π‘‘π‘–π‘šπ‘Žπ‘‘π‘’π‘‘ )2+πœ–β€–π’‚π’Œβ€–)

data term regularization term

local window local values

[Levin et al. ’08]

Page 54: Chap. 10 computational photography

3. Eliminate to get a final energy

Page 55: Chap. 10 computational photography

4.4 Smoke, shadow, and flash matting

・ Smoke matting[Chuang et al. ’02]

input frameremoving the foreground

object

estimated alpha matte

insertion of new objects

Page 56: Chap. 10 computational photography

・ Shadow matting[Chuang et al. ’03]

Page 57: Chap. 10 computational photography

4.5 Video matting

[Chuang et al. ’02]

Page 58: Chap. 10 computational photography

5. Texture analysis and synthesis

Page 59: Chap. 10 computational photography

Texture synthesis is …

Small patch

Similar-looking larger patch

synthesize

Page 60: Chap. 10 computational photography

Texture synthesis using non-parametric sampling

Texture synthesis using non-parametric sampling

[Efros et al. ’99]

[Efros et al. ’01]

Page 61: Chap. 10 computational photography

5.1 hole filling and inpainting

inpaint

Original Inpainted result

Page 62: Chap. 10 computational photography

・ Exemplar-based inpainting[Criminisi et al. ’04]

1. Compute patch priorities

𝑃 (𝒑 )=𝐢 (𝒑 )𝐷 (𝒑 )

C (𝒑 )=βˆ‘

π’’βˆˆΞ¨ π’‘βˆ© ( π‘°βˆ’π›€ )

𝐢 (𝒒)

|Ξ¨ 𝒑|

𝐷 (𝒑 )=|𝛻 𝐼𝑝

βŠ₯βˆ™π’π’‘|𝛼

data termconfidence term

Page 63: Chap. 10 computational photography

2. Find the most similar patch to with top priorityΨ �̂�=arg min

Ξ¨ π’’βˆˆΞ¦π‘‘ (Ξ¨ οΏ½Μ‚οΏ½ ,Ξ¨ 𝒒 )

3. Update confidence values𝐢 (𝒑 )=𝐢 (οΏ½Μ‚οΏ½ )βˆ€ π’‘βˆˆΞ¨ οΏ½Μ‚οΏ½βˆ©Ξ©

Page 64: Chap. 10 computational photography

onio

n p

eel

[Cri

min

isi et

al.

’04

]

original removed region

Remain the gradient alongthe region boundary

Page 65: Chap. 10 computational photography

5.2 Non-photorealistic rendering

Non-photorealistic rendering using texture synthesis

・ Texture transfer・ Image analogies

Page 66: Chap. 10 computational photography

・ Texture transfer

[Efros et al. ’01]

input outputtexture transfer

input outputtexture transfer

Page 67: Chap. 10 computational photography

1. Find the most similar block 𝐸𝑖=π›ΌπΈπ‘œπ‘£π‘’π‘Ÿπ‘™π‘Žπ‘+(1βˆ’π›Ό ) 𝐸 π‘™π‘’π‘šπ‘–π‘›π‘Žπ‘›π‘π‘’

𝐸 𝑗

𝐸𝑖

𝐡 𝑗

𝐡𝑖

( means ordinary texture synthesis. )

Page 68: Chap. 10 computational photography

2. Find the minimum error boundary cut𝑒=(𝐡1

π‘œπ‘£  βˆ’𝐡2π‘œπ‘£   )2

𝐡1π‘œπ‘£ ,𝐡2

π‘œπ‘£

( the minimal cost path corresponds to the valley in the surface . )

Page 69: Chap. 10 computational photography

・ Image analogies

[Hertzmann et al. ’01]

𝐴 𝐴 β€² 𝐡 𝐡 β€²

?: :ΒΏ: :ΒΏ

synthesize this image

Page 70: Chap. 10 computational photography

𝐴 𝐴 β€²

𝐡 𝐡 β€²

unfiltered example NPR-filtered example

target result

Page 71: Chap. 10 computational photography

𝐴 𝐴 β€²

𝐡 𝐡 β€²

unfiltered example filtered example

target result

Page 72: Chap. 10 computational photography

Thank you!