Data-driven methods: Video
Transcript of Data-driven methods: Video
![Page 1: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/1.jpg)
Data-driven methods: Video
15-463: Computational PhotographyAlexei Efros, CMU, Fall 2007
© A.A. Efros
![Page 2: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/2.jpg)
Weather Forecasting for Dummies™Let’s predict weather:
• Given today’s weather only, we want to know tomorrow’s• Suppose weather can only be {Sunny, Cloudy, Raining}
The “Weather Channel” algorithm:• Over a long period of time, record:
– How often S followed by R– How often S followed by S– Etc.
• Compute percentages for each state: – P(R|S), P(S|S), etc.
• Predict the state with highest probability!• It’s a Markov Chain
![Page 3: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/3.jpg)
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
4.04.02.03.03.04.01.06.03.0
Markov Chain
What if we know today and yestarday’s weather?
![Page 4: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/4.jpg)
Text Synthesis
[Shannon,’48] proposed a way to generate English-looking text using N-grams:• Assume a generalized Markov model• Use a large text to compute prob. distributions of
each letter given N-1 previous letters • Starting from a seed repeatedly sample this Markov
chain to generate new letters • Also works for whole words
WE NEED TO EAT CAKE
![Page 5: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/5.jpg)
Mark V. Shaney (Bell Labs)
Results (using alt.singles corpus):• “As I've commented before, really relating to
someone involves standing next to impossible.”
• “One morning I shot an elephant in my arms and kissed him.”
• “I spent an interesting evening recently with a grain of salt”
№
![Page 6: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/6.jpg)
Video TexturesVideo Textures
Arno SchödlRichard Szeliski
David SalesinIrfan Essa
Microsoft Research Georgia Tech
![Page 7: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/7.jpg)
Still photosStill photos
![Page 8: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/8.jpg)
Video clipsVideo clips
![Page 9: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/9.jpg)
Video texturesVideo textures
![Page 10: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/10.jpg)
Problem statementProblem statement
video clip video texture
![Page 11: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/11.jpg)
Our approachOur approach
• How do we find good transitions?
![Page 12: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/12.jpg)
Finding good transitions Finding good transitions
• Compute L2 distance Di, j between all frames
Similar frames make good transitions
frame ivs.
frame j
![Page 13: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/13.jpg)
Markov chain representationMarkov chain representation
2 3 41
Similar frames make good transitions
![Page 14: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/14.jpg)
Transition costs Transition costs
• Transition from i to j if successor of i is similar to j
• Cost function: Ci→j = Di+1, j
• i
j
i+1
j-1
i j→ Di+1, j
![Page 15: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/15.jpg)
Transition probabilitiesTransition probabilities
•Probability for transition Pi→j inversely related to cost:
•Pi→j ~ exp ( – Ci→j / σ2 )
high σ low σ
![Page 16: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/16.jpg)
Preserving dynamicsPreserving dynamics
![Page 17: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/17.jpg)
Preserving dynamics Preserving dynamics
![Page 18: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/18.jpg)
Preserving dynamics Preserving dynamics
• Cost for transition i→j
• Ci→j = wk Di+k+1, j+kΣk = -N
N-1
i
j j+1
i+1 i+2
j-1j-2
i j→Di, j-1 D Di+1, j i+2, j+1
i-1
Di-1, j-2
![Page 19: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/19.jpg)
Preserving dynamics – effect Preserving dynamics – effect
• Cost for transition i→j
• Ci→j = wk Di+k+1, j+kΣk = -N
N-1
![Page 20: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/20.jpg)
2 3 41
Dead endsDead ends
• No good transition at the end of sequence
![Page 21: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/21.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 22: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/22.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 23: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/23.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 24: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/24.jpg)
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 25: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/25.jpg)
2 3 41
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
• Q-learning
Future costFuture cost
![Page 26: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/26.jpg)
Future cost – effectFuture cost – effect
![Page 27: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/27.jpg)
Finding good loopsFinding good loops
• Alternative to random transitions
• Precompute set of loops up front
![Page 28: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/28.jpg)
Visual discontinuitiesVisual discontinuities
• Problem: Visible “Jumps”
![Page 29: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/29.jpg)
CrossfadingCrossfading
• Solution: Crossfade from one sequence to the other.
Ai-2
Ai-2
Bj-2
15
…
…
3
1 2
2 1
3
4
4 4
4 4
4+ + +
Ai-1
Ai-1/Bj-2 Ai-1/Bj-2 Ai-1/Bj-2
Bj-1
Ai
Bj
Ai+1
Bj+1
Bj+1
![Page 30: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/30.jpg)
Morphing Morphing
• Interpolation task:
A25 B2
5 C15+ +
![Page 31: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/31.jpg)
Morphing Morphing
• Interpolation task:
• Compute correspondencebetween pixels of all frames
A25 B2
5 C15+ +
![Page 32: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/32.jpg)
Morphing Morphing
• Interpolation task:
• Compute correspondence between pixels of all frames
• Interpolate pixel position andcolor in morphed frame
• based on [Shum 2000]
A25 B2
5 C15+ +
![Page 33: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/33.jpg)
Results – crossfading/morphingResults – crossfading/morphing
![Page 34: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/34.jpg)
Results – crossfading/morphingResults – crossfading/morphing
Jump Cut Crossfade Morph
![Page 35: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/35.jpg)
CrossfadingCrossfading
![Page 36: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/36.jpg)
Frequent jump & crossfadingFrequent jump & crossfading
![Page 37: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/37.jpg)
Video portraitVideo portrait
• Useful for web pages
![Page 38: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/38.jpg)
• Combine with IBR techniques
Video portrait – 3DVideo portrait – 3D
![Page 39: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/39.jpg)
Region-based analysisRegion-based analysis
• Divide video up into regions
• Generate a video texture for each region
![Page 40: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/40.jpg)
Automatic region analysisAutomatic region analysis
![Page 41: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/41.jpg)
User selects target frame range
User-controlled video texturesUser-controlled video textures
slow variable fast
![Page 42: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/42.jpg)
Video-based animationVideo-based animation
• Like spritescomputer games
• Extract spritesfrom real video
• Interactively control desired motion
©1985 Nintendo of America Inc.
![Page 43: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/43.jpg)
Video sprite extractionVideo sprite extraction
blue screen mattingand velocity estimation
![Page 44: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/44.jpg)
Ci j = + angle α βCi j→ →
vector tomouse pointer
Similarity term Control term
velocity vector
Animation
Video sprite controlVideo sprite control
• Augmented transition cost:
![Page 45: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/45.jpg)
Fi j→
Fi j→
Fi j→ Fi j→
Fi j→
Fi j→Fi j→
SW
W
NWN
NE
E
SES
Goal
Video sprite controlVideo sprite control
• Need future cost computation
• Precompute future costs for a few angles.
• Switch between precomputed angles according to user input
• [GIT-GVU-00-11]
![Page 46: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/46.jpg)
Interactive fishInteractive fish
![Page 47: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/47.jpg)
Summary Summary
• Video clips → video textures• define Markov process• preserve dynamics• avoid dead-ends• disguise visual discontinuities
![Page 48: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/48.jpg)
Discussion Discussion
• Some things are relatively easy
![Page 49: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/49.jpg)
Discussion Discussion
• Some are hard
![Page 50: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/50.jpg)
A final exampleA final example
![Page 51: Data-driven methods: Video](https://reader033.fdocuments.us/reader033/viewer/2022052106/62880c1e3c85ae6428103379/html5/thumbnails/51.jpg)
Michel Gondry train video
http://youtube.com/watch?v=qUEs1BwVXGA