15-463: Computational Photography Alexei Efros, CMU, Fall...
Transcript of 15-463: Computational Photography Alexei Efros, CMU, Fall...
![Page 1: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/1.jpg)
Data-driven methods: Video & Texture
15-463: Computational PhotographyAlexei Efros, CMU, Fall 2008
© A.A. Efros
![Page 2: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/2.jpg)
Michel Gondry train video
http://youtube.com/watch?v=qUEs1BwVXGA
![Page 3: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/3.jpg)
Weather Forecasting for Dummies™Let’s predict weather:
• Given today’s weather only, we want to know tomorrow’s• Suppose weather can only be {Sunny, Cloudy, Raining}
The “Weather Channel” algorithm:• Over a long period of time, record:
– How often S followed by R– How often S followed by S– Etc.
• Compute percentages for each state: – P(R|S), P(S|S), etc.
• Predict the state with highest probability!• It’s a Markov Chain
![Page 4: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/4.jpg)
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
4.04.02.03.03.04.01.06.03.0
Markov Chain
What if we know today and yestarday’s weather?
![Page 5: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/5.jpg)
Text Synthesis
[Shannon,’48] proposed a way to generate English-looking text using N-grams:• Assume a generalized Markov model• Use a large text to compute prob. distributions of
each letter given N-1 previous letters • Starting from a seed repeatedly sample this Markov
chain to generate new letters • Also works for whole words
WE NEED TO EAT CAKE
![Page 6: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/6.jpg)
Mark V. Shaney (Bell Labs)
Results (using alt.singles corpus):• “As I've commented before, really relating to
someone involves standing next to impossible.”
• “One morning I shot an elephant in my arms and kissed him.”
• “I spent an interesting evening recently with a grain of salt”
№
![Page 7: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/7.jpg)
Video TexturesVideo Textures
Arno SchödlRichard Szeliski
David SalesinIrfan Essa
Microsoft Research Georgia Tech
![Page 8: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/8.jpg)
Still photosStill photos
![Page 9: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/9.jpg)
Video clipsVideo clips
![Page 10: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/10.jpg)
Video texturesVideo textures
![Page 11: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/11.jpg)
Problem statementProblem statement
video clip video texture
![Page 12: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/12.jpg)
Our approachOur approach
• How do we find good transitions?
![Page 13: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/13.jpg)
Finding good transitions Finding good transitions
• Compute L2 distance Di, j between all frames
Similar frames make good transitions
frame ivs.
frame j
![Page 14: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/14.jpg)
Markov chain representationMarkov chain representation
2 3 41
Similar frames make good transitions
![Page 15: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/15.jpg)
Transition costs Transition costs
• Transition from i to j if successor of i is similar to j
• Cost function: Ci→j = Di+1, j
• i
j
i+1
j-1
i j→ Di+1, j
![Page 16: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/16.jpg)
Transition probabilitiesTransition probabilities
•Probability for transition Pi→j inversely related to cost:
•Pi→j ~ exp ( – Ci→j / σ2 )
high σ low σ
![Page 17: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/17.jpg)
Preserving dynamicsPreserving dynamics
![Page 18: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/18.jpg)
Preserving dynamics Preserving dynamics
![Page 19: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/19.jpg)
Preserving dynamics Preserving dynamics
• Cost for transition i→j
• Ci→j = wk Di+k+1, j+kΣk = -N
N-1
i
j j+1
i+1 i+2
j-1j-2
i j→Di, j-1 D Di+1, j i+2, j+1
i-1
Di-1, j-2
![Page 20: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/20.jpg)
Preserving dynamics – effect Preserving dynamics – effect
• Cost for transition i→j
• Ci→j = wk Di+k+1, j+kΣk = -N
N-1
![Page 21: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/21.jpg)
2 3 41
Dead endsDead ends
• No good transition at the end of sequence
![Page 22: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/22.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 23: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/23.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 24: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/24.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 25: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/25.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 26: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/26.jpg)
2 3 41
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
• Q-learning
Future costFuture cost
![Page 27: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/27.jpg)
Future cost – effectFuture cost – effect
![Page 28: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/28.jpg)
Finding good loopsFinding good loops
• Alternative to random transitions
• Precompute set of loops up front
![Page 29: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/29.jpg)
Video portraitVideo portrait
• Useful for web pages
![Page 30: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/30.jpg)
Region-based analysisRegion-based analysis
• Divide video up into regions
• Generate a video texture for each region
![Page 31: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/31.jpg)
Automatic region analysisAutomatic region analysis
![Page 32: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/32.jpg)
User selects target frame range
User-controlled video texturesUser-controlled video textures
slow variable fast
![Page 33: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/33.jpg)
Video-based animationVideo-based animation
• Like spritescomputer games
• Extract spritesfrom real video
• Interactively control desired motion
©1985 Nintendo of America Inc.
![Page 34: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/34.jpg)
Video sprite extractionVideo sprite extraction
blue screen mattingand velocity estimation
![Page 35: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/35.jpg)
Ci j = + angle α βCi j→ →
vector tomouse pointer
Similarity term Control term
velocity vector
Animation
Video sprite controlVideo sprite control
• Augmented transition cost:
![Page 36: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/36.jpg)
Fi j→
Fi j→
Fi j→ Fi j→
Fi j→
Fi j→Fi j→
SW
W
NWN
NE
E
SES
Goal
Video sprite controlVideo sprite control
• Need future cost computation
• Precompute future costs for a few angles.
• Switch between precomputed angles according to user input
• [GIT-GVU-00-11]
![Page 37: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/37.jpg)
Interactive fishInteractive fish
![Page 38: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/38.jpg)
Summary Summary
• Video clips → video textures• define Markov process• preserve dynamics• avoid dead-ends• disguise visual discontinuities
![Page 39: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/39.jpg)
Discussion Discussion
• Some things are relatively easy
![Page 40: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/40.jpg)
Discussion Discussion
• Some are hard
![Page 41: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/41.jpg)
“Amateur” by Lasse Gjertsen
http://www.youtube.com/watch?v=JzqumbhfxRo
![Page 42: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/42.jpg)
Texture
• Texture depicts spatially repeating patterns• Many natural phenomena are textures
radishes rocks yogurt
![Page 43: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/43.jpg)
Texture Synthesis• Goal of Texture Synthesis: create new samples of
a given texture• Many applications: virtual environments, hole-
filling, texturing surfaces
![Page 44: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/44.jpg)
The Challenge
• Need to model the whole spectrum: from repeated to stochastic texture
repeated
stochastic
Both?
![Page 45: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/45.jpg)
Efros & Leung Algorithm
• Assuming Markov property, compute P(p|N(p))– Building explicit probability tables infeasible
pp
Synthesizing a pixel
non-parametricsampling
Input image
– Instead, we search the input image for all similar neighborhoods — that’s our pdf for p
– To sample from this pdf, just pick one match at random
![Page 46: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/46.jpg)
Some Details• Growing is in “onion skin” order
– Within each “layer”, pixels with most neighbors are synthesized first
– If no close match can be found, the pixel is not synthesized until the end
• Using Gaussian-weighted SSD is very important– to make sure the new pixel agrees with its closest
neighbors– Approximates reduction to a smaller neighborhood
window if data is too sparse
![Page 47: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/47.jpg)
Neighborhood Window
input
![Page 48: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/48.jpg)
Varying Window Size
Increasing window size
![Page 49: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/49.jpg)
Synthesis Resultsfrench canvas rafia weave
![Page 50: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/50.jpg)
More Resultswhite bread brick wall
![Page 51: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/51.jpg)
Homage to Shannon
![Page 52: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/52.jpg)
Hole Filling
![Page 53: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/53.jpg)
Extrapolation
![Page 54: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/54.jpg)
Summary• The Efros & Leung algorithm
– Very simple– Surprisingly good results– Synthesis is easier than analysis!– …but very slow
![Page 55: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/55.jpg)
pp
Image Quilting [Efros & Freeman]
• Observation: neighbor pixels are highly correlated
Input image
non-parametricsampling
BB
Idea:Idea: unit of synthesis = blockunit of synthesis = block• Exactly the same but now we want P(B|N(B))
• Much faster: synthesize all pixels in a block at once
• Not the same as multi-scale!
Synthesizing a block
![Page 56: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/56.jpg)
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
![Page 57: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/57.jpg)
min. error boundary
Minimal error boundaryoverlapping blocks vertical boundary
__ ==22
overlap error
![Page 58: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/58.jpg)
Our Philosophy
• The “Corrupt Professor’s Algorithm”:– Plagiarize as much of the source image as you can– Then try to cover up the evidence
• Rationale: – Texture blocks are by definition correct samples of
texture so problem only connecting them together
![Page 59: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/59.jpg)
![Page 60: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/60.jpg)
![Page 61: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/61.jpg)
![Page 62: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/62.jpg)
![Page 63: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/63.jpg)
![Page 64: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/64.jpg)
![Page 65: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/65.jpg)
Failures(ChernobylHarvest)
![Page 66: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/66.jpg)
input image
Portilla & Simoncelli
Wei & Levoy Our algorithm
Xu, Guo & Shum
![Page 67: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/67.jpg)
Portilla & Simoncelli
Wei & Levoy Our algorithm
Xu, Guo & Shum
input image
![Page 68: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/68.jpg)
Portilla & Simoncelli
Wei & Levoy Our algorithm
input image
Xu, Guo & Shum
![Page 69: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/69.jpg)
Political Texture Synthesis!
![Page 70: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/70.jpg)
Fill Order
• In what order should we fill the pixels?
![Page 71: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/71.jpg)
Fill Order
• In what order should we fill the pixels?– choose pixels that have more neighbors filled– choose pixels that are continuations of
lines/curves/edgesCriminisi, Perez, and Toyama. “Object Removal by Exemplar-based Inpainting,” Proc. CVPR, 2003.
![Page 72: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/72.jpg)
Exemplar-based Inpainting demo
http://research.microsoft.com/vision/cambridge/i3l/patchworks.htm
![Page 73: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/73.jpg)
++ ==
Application: Texture Transfer
• Try to explain one object with bits and pieces of another object:
![Page 74: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/74.jpg)
Texture Transfer Constraint
Texture sample
![Page 75: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/75.jpg)
• Take the texture from one image and “paint” it onto another object
Texture Transfer
Same as texture synthesis, except an additional constraint:1. Consistency of texture 2. Similarity to the image being “explained”
![Page 76: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/76.jpg)
==++
![Page 77: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/77.jpg)
![Page 78: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/78.jpg)
Image Analogies
Aaron Hertzmann1,2
Chuck Jacobs2
Nuria Oliver2
Brian Curless3
David Salesin2,3
1New York University2Microsoft Research3University of Washington
![Page 79: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/79.jpg)
Image Analogies
A A’
B B’
![Page 80: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/80.jpg)
![Page 81: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/81.jpg)
Blur Filter
![Page 82: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/82.jpg)
Edge Filter
![Page 83: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/83.jpg)
A A’
B B’
Artistic Filters
![Page 84: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/84.jpg)
Colorization
![Page 85: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/85.jpg)
Texture-by-numbers
A A’
B B’
![Page 86: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/86.jpg)
Super-resolution
A A’
![Page 87: 15-463: Computational Photography Alexei Efros, CMU, Fall 2008graphics.cs.cmu.edu/courses/15-463/2008_fall/... · Text Synthesis [Shannon,’48] proposed a way to generate English-looking](https://reader033.fdocuments.us/reader033/viewer/2022050110/5f48406eb3acf43dfc1697b9/html5/thumbnails/87.jpg)
Super-resolution (result!)
B B’