Texture Optimization for Example-based Synthesis
42
Texture Optimization for Example-based Synthesis Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra
description
Texture Optimization for Example-based Synthesis. Vivek Kwatra Irfan Essa Aaron Bobick Nipun Kwatra. Image vs. Texture. Example-based Texture Synthesis. Input Example. Related Work. Local region-growing method -Pixel-based - Patch-based Global optimization-based method - PowerPoint PPT Presentation
Transcript of Texture Optimization for Example-based Synthesis
Texture Optimization for Example-based Synthesis
Vivek Kwatra
Irfan EssaAaron BobickNipun Kwatra
Image vs. Texture
Example-based Texture Synthesis
Input Example
Related Work
• Local region-growing method
-Pixel-based
-Patch-based
• Global optimization-based method Heeger and Bergen sig95,Pyramid-based texture synthesis
Paget and Longstaff IEEE Tran… 98,Texture synthesis via a noncausal nonparametric multiscale markov random field
…..
• Physical Simulation et al
Example-based Texture Synthesis
1. Dynamic texturing (animation/video)
2. Greater control: external knobs, e.g., flow,
shape, illumination
3. Texture quality evaluation and refinement
Motivation
+
InputTexture
InputFlow Field
OutputSequence
Limitations of previous work
• Pixel-based
can not maintain global structure of the texture
inconsistencies
• Patch-based
have no control over individual pixel values
• Global methods:
pixel-to-pixel interactions
insufficient to capture large scale structures
complex formulations difficult to optimize
Texture Energy
• Pixel-by-pixel comparison of source and target not possible
• Compare texture “elements”– Local pixel neighborhoods
• Want each target neighborhood to be similar to some source neighborhood
Texture Energy
X(target frame)
Z(source)
Energy for Single Neighborhood
p (pixel)Z(source)
X(target frame)
Energy for Single Neighborhood
Xp (neighborhood)Z(source)
X(target frame)
Energy for Single Neighborhood
Xp (neighborhood)Z(source)
X(target frame)
Energy for Single Neighborhood
(nearest neighbor) Zp
Xp (neighborhood)Z(source)
X(target frame)
Energy for Single Neighborhood
X
Z
(nearest neighbor) Zp
Xp (neighborhood)
Texture Energy(single neighborhood)
2
t P P PE X X Z
Texture Energy for Entire Image
X
Z
Texture Energy for Entire Image
X
Z
individual neighborhood energy
2;t P P P
P X X
E X Z X Z
4.Repeat iteratively until set {ZP} stops changing
3.Update X to be texture that minimizes the energy
Texture Synthesis
1.Initialize texture X
2
t P P PE X X Z
2.Find closest input neighborhood ZP corresponding to each output neighborhood XP
2;t P P P
P X X
E X Z X Z
Texture Synthesis
Gradient-based Energy
D is the differentiation operator and µ is a relative weighting coefficient
Muti-level Synthesis
• Multi-resolution
• Multi-scale/multiple neighborhood sizes
Reasons a good initialization of the texture avoid undesirable local minima avoid inconsistencies capture large scale structures
• Random Initialization
• Multiple– Resolution Levels– Neighborhood
Sizes
• Progressively refined output
Texture Energy Graph
Level 1
Level 2
Level 3
Level 4
Level 5
Level 6
Level 7
0 5 10 15 20 25 30 35 40 45 500
50
100
150
200
250
300
350
400
Number of Iterations
Controllable Synthesis
2; c c
ck
E X X X k X k
; ; ct P cE X E X Z E X X
• Flow consistency– Perceived motion similar to flow
• Texture similarity– Shape, size, orientation of texture
elements similar to source
Flow-guided Texture Animation
Target FlowFlowing Target
Source Texture
TextureSimilarity
FlowConsistency
Naive Approach
Target Flow
• Ignore Texture Similarity
• Warp each frame via flow field
X1 Xn
TargetFrames X0
Warp
Flowing Target
FlowConsistency
Source Texture
TextureSimilarity
Source Texture
TextureSimilarity
Naive Approach
Texture structure not maintained
Target FlowFlowing Target
FlowConsistency
Source Texture
TextureSimilarity
Warp
Instead of…
WarpXnX0 X1
TargetFrames
Warp
Instead of…
WarpXnX0 X1
TargetFrames
Warp + Correct
“Correct” Operation
TargetFrames XX–
Warp
W
Correct
WarpedFrame
SynthesizedFrame
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Warp
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Flow ConsistencyWarp
Link to original goals
“Correct” Operation
XTargetFrames X–
W
Correct
Source
Flow Consistency
Texture Similarity
Warp
Energy = Flow Energy + Texture Energy
Optimization of Energy
XTargetFrames X–
W
Correct
Source
Flow Consistency
Texture Similarity
Warp
Flow Energy
1 2 1, , , Lf f f f
2D flow field
:if frame i frame i+1
pixel (p,i) pixel (q,i+1)
2
1: 1 ,
; , , 1 ,i
c ii L p i X
E X f X p i X q i q f p
Results: Texture Synthesis
Results: Texture Synthesis
Input Wei-Levoy[Wei’00]
Image Quilting[Efros’01]
Graph-cuts[Kwatra’03]
Texture Optimization
[Kwatra’05]
Time Cost
• Static texture synthesis– 5-10 iterations per resolution level
• Flowing texture synthesis– 20-60 seconds per frame
Resolution One iteration Total time
64x64 2.5 sec 20 sec
128x128 10 sec 2 min
256x256 25 sec 8.5 min
Limitations
• Optimization finds local minima
• Blurry / misaligned texture elements
• Texture scale should be compatible
with flow field
Thanks a lot!
Pixel-based Methods
• Compare local causal neighbourhoods Efros and Leung (ICCV ’99)
Wei and Levoy (Siggraph 2000)Ashikhmin (I3D 2001)
Input Output
Patch-based Methods
• Copy patches of pixels rather than single pixels
Chaos Mosaic, Xu et al, 1997
Patch-Based Sampling, Liang et al(ACM 2001)
Image Quilting, Efros and Williams(Siggraph 2001)
Back
