Texture Optimization for Example-based Synthesis
-
Upload
rhonda-cook -
Category
Documents
-
view
43 -
download
0
description
Transcript of Texture Optimization for Example-based Synthesis
![Page 1: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/1.jpg)
Texture Optimization for Example-based Synthesis
Vivek Kwatra1,2
Irfan Essa1
Aaron Bobick1
Nipun Kwatra1
1GVU Center/College of Computing, Georgia Tech2Computer Science Dept., UNC Chapel Hill
![Page 2: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/2.jpg)
Motivation
+ =
InputTexture
InputFlow Field
OutputSequence
![Page 3: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/3.jpg)
Example-based Texture Synthesis
![Page 4: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/4.jpg)
Example-based Texture Synthesis
1. Dynamic texturing (animation/video)
2. Greater control: external knobs, e.g., flow, shape, illumination
3. Explicit texture quality evaluation and refinement
![Page 5: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/5.jpg)
Contributions I
• Global Optimization for texture synthesis– Global texture quality metric
– Iterative multi-scale optimization
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
Level 7
0 5 10 15 20 25 30 35 40 45 500
50
100
150
200
250
300
350
400
Number of Iterations
![Page 6: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/6.jpg)
Contributions II
• Flow-guided texture animation– Flow incorporated into optimization
– Applications: fluid rendering, flow visualization
+ =
InputTexture
InputFlow Field
OutputSequence
![Page 7: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/7.jpg)
Related Work (a selected sample)Lo
cal v
s. G
loba
l
Pixel vs. Patch
Non-parametricsampling
[Efros’99,Wei’00]Image Quilting
[Efros’01]
TextureOptimization
[Kwatra’05]
Histogram Matching[Heeger’95]
MRF Annealing[Paget’98]
Graphcut Textures[Kwatra’03]
![Page 8: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/8.jpg)
Related Work (a selected sample)Lo
cal v
s. G
loba
l
Pixel vs. Patch
Non-parametricsampling
[Efros’99,Wei’00]Image Quilting
[Efros’01]
TextureOptimization
[Kwatra’05]
Histogram Matching[Heeger’95]
MRF Annealing[Paget’98]
Better Quality
Graphcut Textures[Kwatra’03]
![Page 9: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/9.jpg)
Related Work (a selected sample)Lo
cal v
s. G
loba
l
Pixel vs. Patch
Non-parametricsampling
[Efros’99,Wei’00]Image Quilting
[Efros’01]
TextureOptimization
[Kwatra’05]
Histogram Matching[Heeger’95]
MRF Annealing[Paget’98]
Better Quality
Greater Controllability
Graphcut Textures[Kwatra’03]
![Page 10: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/10.jpg)
Related Work
• Flow-guided Synthesis– Advected Textures [Neyret’03]
– Flow-based Video Editing [Bhat’04]
• Optimization– Image-based Priors [Fitzgibbon’03]
– Space-time Video Completion [Wexler’04]
![Page 11: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/11.jpg)
• Flow consistency– Perceived motion similar to flow
• Texture similarity– Shape, size, orientation of texture
elements similar to source
Flow-guided Texture Animation
Target FlowFlowing Target
Source Texture
TextureSimilarity
FlowConsistency
![Page 12: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/12.jpg)
Naive Approach
Target Flow
• Ignore Texture Similarity
• Warp each frame via flow field
X1 Xn
TargetFrames X0
Warp
Flowing Target
FlowConsistency
Source Texture
TextureSimilarity
Source Texture
TextureSimilarity
![Page 13: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/13.jpg)
Naive Approach
Texture structure not maintained
Target FlowFlowing Target
FlowConsistency
Source Texture
TextureSimilarity
![Page 14: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/14.jpg)
Warp
Instead of…
WarpXnX0 X1
TargetFrames
![Page 15: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/15.jpg)
Warp
Instead of…
WarpXnX0 X1
TargetFrames
![Page 16: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/16.jpg)
Warp + Correct
Our Approach
TargetFrames XX–
Warp
W
Correct
WarpedFrame
SynthesizedFrame
![Page 17: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/17.jpg)
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Warp
![Page 18: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/18.jpg)
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Flow ConsistencyWarp
![Page 19: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/19.jpg)
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Source
Flow Consistency
Texture Similarity
Warp
![Page 20: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/20.jpg)
Energy = Flow Energy + Texture Energy
Optimization of Energy / Cost
XTargetFrames X–
W
Correct
Source
Flow Consistency
Texture Similarity
Warp
![Page 21: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/21.jpg)
Flow Energy
XTargetFrames X–
Warp
W
Flow Energy
![Page 22: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/22.jpg)
Texture Energy
• Tricky!– Pixel-by-pixel comparison of source and target
not possible
• Compare texture “elements”– Local pixel neighborhoods
• Want each target neighborhood to be similar to some source neighborhood
![Page 23: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/23.jpg)
Texture Energy
X(target frame)
Z(source)
![Page 24: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/24.jpg)
Energy for Single Neighborhood
p (pixel)Z(source)
X(target frame)
![Page 25: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/25.jpg)
Energy for Single Neighborhood
Xp (neighborhood)Z(source)
X(target frame)
![Page 26: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/26.jpg)
Energy for Single Neighborhood
Xp (neighborhood)Z(source)
X(target frame)
![Page 27: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/27.jpg)
Energy for Single Neighborhood
(nearest neighbor) Zp
Xp (neighborhood)Z(source)
X(target frame)
![Page 28: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/28.jpg)
Energy for Single Neighborhood
X
Z
(nearest neighbor) Zp
Xp (neighborhood)
Texture Energy(single neighborhood)
![Page 29: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/29.jpg)
Texture Energy for Entire Image
X
Z
![Page 30: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/30.jpg)
Texture Energy for Entire Image
X
Z
individual neighborhood energy
![Page 31: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/31.jpg)
• Optimize Total Energy of target frame:
• Initialize: X W(target frame warped frame)
• Iteratively improve target frame
Optimization
![Page 32: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/32.jpg)
Optimization: Intuition
Warped Frame
Source
Target Frame
Flow Force
Texture ForceZp
Xp
Wp
![Page 33: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/33.jpg)
Optimization: Intuition
Source
Zp
Xp
Wp
Blend
Warped Frame
Target Frame
![Page 34: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/34.jpg)
Overlapping Neighborhoods
Source
Wp
BlendXqZq
Xp
Warped Frame
Target Frame
Zp
![Page 35: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/35.jpg)
We are not done yet!
Source
Xp
Wp
ZpNearest neighbor may change
Warped Frame
Target Frame
![Page 36: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/36.jpg)
Iterative Algorithm
• Step 1 – Find Nearest Source Neighborhoods
• Step 2– Blend Source & Warped Neighborhoods
• Repeat until convergence
![Page 37: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/37.jpg)
Iterative Algorithm
• Step 1 – Find Nearest Source Neighborhoods
• Minimize E( X ) w.r.t {Zp}
• Approximate nearest-neighbors– Hierarchical Clustering [Johnson’67] – Tree-Structured Vector Quantization [Wei’00]
![Page 38: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/38.jpg)
Iterative Algorithm
• Step 2– Blend Source & Warped Neighborhoods
• Minimize E( X ) w.r.t X
• Set Solve Linear System
![Page 39: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/39.jpg)
Robust Optimization
• Quadratic Texture Energy Et sensitive to outlier neighborhoods
• Non-quadratic Et more robust
– Et( X ) = p kXp – Zpkr, 0 < r < 2
![Page 40: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/40.jpg)
Robust Optimization
• Quadratic Texture Energy Et sensitive to outlier neighborhoods
• Non-quadratic Et more robust
– Et( X ) = p kXp – Zpkr, 0 < r < 2
• Solve weighted quadratic at each iteration Et( X ) = p kXp – Zpkr-2kXp – Zpk2
= p p kXp – Zpk2
![Page 41: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/41.jpg)
Minimize Texture Energy Alone
• Novel Static Texture Synthesis Algorithm
CorrectXX–
W
Texture Similarity
Flow Consistency
Source
![Page 42: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/42.jpg)
• Random Initialization
• Multiple– Resolution Levels– Neighborhood Sizes
• Progressively refined output
Texture Energy Graph
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
Level 7
0 5 10 15 20 25 30 35 40 45 500
50
100
150
200
250
300
350
400
Number of Iterations
![Page 43: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/43.jpg)
• Random Initialization
• Multiple– Resolution Levels– Neighborhood Sizes
• Progressively refined output
Texture Energy Graph
![Page 44: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/44.jpg)
Results: Texture Synthesis
![Page 45: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/45.jpg)
![Page 46: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/46.jpg)
Results: Texture Synthesis
Input Wei-Levoy[Wei’00]
Image Quilting[Efros’01]
Graph-cuts[Kwatra’03]
Texture Optimization
[Kwatra’05]
![Page 47: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/47.jpg)
Results: Texture Synthesis
Image Quilting[Efros’01]
Graph-cuts[Kwatra’03]
Texture Optimization[Kwatra’05]
![Page 48: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/48.jpg)
Results: Flow-guided Synthesis
![Page 49: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/49.jpg)
Results: Timing
• Static texture synthesis– 5-10 iterations per resolution level
• Flowing texture synthesis– 20-60 seconds per frame
Resolution One iteration Total time
64x64 2.5 sec 20 sec
128x128 10 sec 2 min
256x256 25 sec 8.5 min
![Page 50: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/50.jpg)
Summary
• Global Optimization for controllable texture synthesis
• Explicit evaluation of texture quality
• Iterative progressive refinement of texture
• Flow-guided texture animation within optimization framework
+ =Input
TextureInput
Flow FieldOutput
Sequence
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
Level 7
0 5 10 15 20 25 30 35 40 45 500
50
100
150
200
250
300
350
400
Number of Iterations
![Page 51: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/51.jpg)
Limitations and Future Work
• Optimization finds local minima– Blurry / misaligned texture elements
• Texture scale should be compatible with flow field
• Extensions– Controllable video textures
– Other control criteria like shape, illumination
![Page 52: Texture Optimization for Example-based Synthesis](https://reader036.fdocuments.us/reader036/viewer/2022062321/56812f9c550346895d95195d/html5/thumbnails/52.jpg)
Acknowledgements
• Eugene Zhang for flow design code
• Greg Turk, Gabriel Brostow, Delphine Nain, Ravi Ruddarraju, Gaurav Chanda, Stephanie Brubaker, Mitch Parry, Pei Yen
• Computational Perception Lab, Geometry Group @ Georgia Tech
• CMU Graphics Lab
Web page:
http://www.cc.gatech.edu/cpl/projects/textureoptimization/