Hybrid Texture Synthesis
-
Upload
cole-lynch -
Category
Documents
-
view
46 -
download
0
description
Transcript of Hybrid Texture Synthesis
Hybrid Texture SynthesisHybrid Texture Synthesis
Andrew NealenAndrew NealenMarc AlexaMarc Alexa
Discrete Geometric Modeling Group Discrete Geometric Modeling Group (DGM)(DGM)
Technische Universität DarmstadtTechnische Universität Darmstadt
Eurographics Symposium on Rendering 2003
Outline
Introduction
2D Texture Synthesis
nxm Input Texture
NxM Output Texture
► The goal: Synthesize an output texture which is perceptually similar to the input texture. Also ensure that the result contains sufficient variation.
Introduction
Existing (and Impressive) Technology Pixel-Based:
Non-parametric Sampling [Efros and Leung 1999]
Tree-structured Vector Quantization [Wei and Levoy 2000]
Image Analogies [Hertzmann et al. 2001] Patch-Based:
Patch-Based Sampling [Guo et al. 2001] Image Quilting [Efros and Freeman 2001]
Introduction
Possible Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring
Wei/Levoy Algorithm:Can occur when using structured Textures with rich histograms.This is due to the L2 norm.
Introduction
Possible Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring
Image Quilting Algorithm:boundary mismatch artifacts are noticeable
Introduction
Possible Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring
Patch-Based Sampling Algorithm:Can occur when texture features are not well-aligned across patch boundaries
Hybrid Texture Synthesis Combine best aspects from other approaches
while avoiding (or improving over) known pitfalls: Patch-based algorithms
Are good at preserving global structure Can introduce artifacts along patch boundaries
Pixel-based algorithms Preserve local coherence (MRF model: local and
stationary) Possibly fail to preserve global structure This is especially problematic for textures with rich
histograms and many high frequency features, due to the smoothing nature of the distance metric (L2 norm)
Hybrid Texture Synthesis (HTS)
Hybrid, two-fold approach to texture synthesis Adaptive patch sampling [Soler et al. 2002]
Start with a uniform quadrilateral grid of patches (the output)
Recursively split a patch if the best fit taken from the input texture is not good enough (a user-defined tradeoff max)
Overlap re-synthesis (novel overlap repair strategy) Mark invalid pixels in the overlap region Validity map is defined by the error surface E(x)=(xsrc –
xdst)2 and a user-defined tolerance parameter max
Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy
HybridSynthesize Algorithm1: HYBRIDSYNTHESIZE(T,P,ov,Δmax,δmax,R) : R2: for all patches pi ∈P do3: [Pi,Δi]← FINDBESTPATCH(T,Ri−1, pi,ov)4: if (Δi < Δmax or ISSINGLEPIXEL(pi)) then5: Si ← ERRORSURFACE(Pi,Ri−1)6: Pi ← MARKINVALIDPIXELS(Pi,Si,δmax)7: Mi ← BUILDTRAVERSALMAP(Pi)8: Ri,composite ← COMPOSE(Pi,Ri−1)9: Ri ← OVERLAPRESYNTHESIS(T,Ri,composite,Mi)10: else11: Ps ← SPLITPATCH(pi)12: ovs ←max(3, ov/2)13: Ri ← HYBRIDSYNTHESIZE(T,Ps,ovs,Δmax,δmax,Ri−1)
14: end if15: end for16: return R
FindBestPatch(Algo.)
1: FINDBESTPATCH(T,Ri−1, pi,ov) : [Pi,Δi]2: Ii, Ji ← BUILDIMAGEMASK(Ri−1, pi,ov)3: Ei ← ERRORIMAGE(T, Ii, Ji)4: Ei,trim ← TRIMINVALIDREGIONS(Ei, Ji, pi)5: [Pi,Δi]← BESTPATCH(Ei,trim,T)6: return [Pi,Δi]
In Search of Good PatchesSynthesizing a single patch
Synthesize Black Patch i
In Search of Good PatchesExtracting the image mask
Synthesize Black Patch i
Grow patch by overlap (toroidally)
In Search of Good PatchesExtracting the image mask
Image Mask (Ii)Synthesize Black Patch i Binary Support (Ji)
Extract image mask (Ii) and binary support (Ji), and circularly shift to upper left corner
In Search of Good PatchesComputing the error image
Image Mask (Ii)Input Texture (T) Binary Support (Ji)
* Compute error Ei(x0) between Ii and T for each circular shift x0=(x,y) of the input texture T
In Search of Good PatchesComputing the error image
Error Image (Ei)Input Texture (T)
x
xxxxxc
cicicii
i TIwJκ
E 20,,0 ))()(()(
1)(
C={R,G,B} (both with RGB color values in [0,1])
WR,G,B = {0.299,0.587,0.114}
X0: circular shift of T
)(xJκixi
(1)
In Search of Good PatchesSelecting a Patch
Error Image (E)Input Texture (T) Selected Patch (P)
)))((())((2)(1
)( 02
0,2
,0 XTJXTIIWκ
E ciccicic
ci
ix
xx (2)
Given that Ji(x) Ii(x) = Ii(x),and the cross correlation between two images (functions) f ◇ g is defined as ( f ◇ g )(x0) = Σx f (x)g(x+x0)The correlation f ◇ g between two functions can be computed in O(N logN)in fourier space as f ◇ g = F-1(F( f )∗F(g))(N :number of pixels in the input texture)In implementation it’s can be pre-compute the fourier transform for T and need only recompute the fourier transforms of Ii and Ji for each new patch pi.
In Search of Good PatchesSelecting a Patch
Error Image (E)Input Texture (T) Selected Patch (P)
Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n) per patch.
Adaptive Patch SamplingPrinciple
Essentially: Hierarchical Pattern Mapping applied to the plane [Soler et al. 2002]
i > max Split PatchSynthesize White Patch i < max
Adaptive Patch SamplingTrade-off
Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Adaptive Patch SamplingTrade-off
Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Adaptive Patch SamplingTrade-off
Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Overlap Re-synthesisOverlap error and pixel invalidation
Patch and Image Mask Selected Patch (P)
Overlap Re-synthesisOverlap error and pixel invalidation
Image Mask (I) Selected Patch (P)
c
cicicii PIwJS 2,, ))()(()()( xxxx
Overlap Re-synthesisOverlap error and pixel invalidation
Image Mask (I) Selected Patch (P)Error Surface (S)
c
cicicii PIwJS 2,, ))()(()()( xxxx
Overlap Re-synthesisOverlap error and pixel invalidation
Patch and Image Mask P with Invalid Pixels (blue)Error Surface (S)
If(Ji(x) !=0 and Si(x)> δmax )
as invalid and in need of per-pixel re-synthesis.else if( Ji(x) != 0 and Si(x) < δmax )are preserved and a stepwise alpha mask (feathering) is applied during Compositing.
(Trade-off: user-defined max is the pixel error tolerance. Setting to 1 results in pure feathering.)
Overlap Re-synthesisCompositing
Intermediate Result Patch with Invalid Pixels
P OVER Ri-1
Overlap Re-synthesisCompositing
Composited Result Patch with Invalid Pixels
Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixel region
Solution: alternative ordering (Pixel Traversal Map)
Pixel Traversal-Map
1: BUILDTRAVERSALMAP(Pi) : Mi
2: level = 13: Dlevel ←binary image of size Pi initialized to 04: Dlevel ←set all valid pixels ∈ Pi to 15: Mi ←Dlevel
6: while (∃pixel ∈ Dlevel ∧ pixel = 0) do7: level = level+18: Dlevel ← DILATE(Dlevel−1,Disk)9: Mi ←Mi +(Dlevel ∧Dlevel−1) ∗level10: end while11: return Mi
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform).
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
Pixel Traversal Map: step 1
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
Pixel Traversal Map: step 2 ...
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
Pixel Traversal Map: ... step n
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image.
Overlap Re-synthesisPer-Pixel Re-synthesis
Pixel Traversal Map (M)
Synthesize red pixel from valid (or already re-synthesized) pixels
Intermediate Result
Overlap Re-synthesisPer-Pixel Re-synthesis
Image Mask (Ij)
Binary Support (Jj)
Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel.
Intermediate Result
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result Pixel Traversal Map (M)
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result Shifted
Pixel Traversal Map (M)
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result with Traversal Map
Pixel Traversal Map (M)
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisStep 1
Pixel Traversal Map (M)
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisStep 2
Pixel Traversal Map (M)
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisStep 3
Pixel Traversal Map (M)
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisResult
Pixel Traversal Map (M)
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
ResultsSynthesis Comparisons
Input
PBS HTS
PBS HTS
Input
ResultsSynthesis Comparisons
Input
PBS HTS
PBS HTS
Input
ResultsSynthesis Comparisons
Input
PBS HTS
PBS HTS
Input
Results
presence of high frequency structure cause the error metric (L2 Norm)
Conclusions and Future Work
Improve Computational Complexity Pixel neighborhoods of patch and pixel
stage are not known a priori, so precomputation is not straightforward
We can solve this in the pixel stage by employing k-coherence search [Tong et al. 2002] [Ashikhmin 2001]
Improve Error Metric Still using the L2 norm due to its simplicity Develop a metric which takes feature
mismatch into account