Fast edge-preserving/-aware high dimensional filters for image & video processing
-
Upload
yu-huang -
Category
Technology
-
view
2.183 -
download
1
description
Transcript of Fast edge-preserving/-aware high dimensional filters for image & video processing
Fast Edge-Preserving/Aware Filters for Image/Video Processing
Yu Huang Sunnyvale, California
Outline • Gaussian filter as weighted average • Bilateral filter (Note: anisotropic diffusion filter is equivalent)
– Separable bilateral filtering (range and spatial kernels);
• Fast bilateral filter – Acceleration by piecewise linear approximation;
• Splatting (downsample), blurring (FFT) and slicing (upsample); – Bilateral grid (importance sampling); – Gaussian KD-tree; – Pemutohedral Lattice.
• Fast Gauss (Transform) bilateral filtering; – Constant time bilateral filtering O(1).
• Fast median (distributed histogram) filter • Smooth local histogram filter and its acceleration • Accelerated trilateral filter (gradient preserving) • Domain Transform (DT) as a real-time edge preserving filter • Patch Match • Coherent Sensitivity Hashing (CSH) • Propagation-assisted KD-tree • References
Gaussian Filter as Weighted Average • Weight of x depends on distance to x
),( xxf )(xI
output input
)(xJ x
xx
x
Bilateral Filter [Tomasi’98] • Spatial Gaussian f • Gaussian g on the intensity difference • Edge preserving filter
)(xJ ),( xxf ))()(( xIIg x )(xIx)(
1
xk
output input
x
Cross-Bilateral Filter
q
p
output input
p
S
IIIGGW
IBFq
qqp
p
p qp ||||||1
][rs
[Eisemann’04] [Petschnigg’04]
Edge Preserving vs. Edge Blurring
Input Image Edge-Preserving Base Image Edge-Blurring Base Image
Edge-Preserving Enhanced Image Edge-Blurring Enhanced Image
Halo Artifacts
Anisotropic Diffusion Filter
c(p, t) is large when p is not a part of an edge c(p, t) is small when p is a part of an edge
• Diffusion [Perona & Malik 1990]: – Diffusion is smoothing: pixel ;
– Anisotropic kernel preserve edge: strong edge reduces kernel size;
• Equivalent to bilateral filter, but not easy to accelerate it.
I
Separable Bilateral Filtering [Pham’05]
Pros: extremely fast (fixed spatial weight + LUT for tonal weight)
Cons: effective filtering kernel is slightly distorted
Acceleration of Bilateral Filters • Discretize the set of possible I(x) – sparse in intensity; • Piecewise linear approximation;
• Perform linear Gaussian blur (FFT); • Sub-sampling in spatial domain; • Linear interpolation in between.
• k(x) is treated similarly • [Durand, Siggraph 2002]
)(xJ ),( xxf ))()(( xIIg x )(xIx)(
1
xk
higher dimensional functions
Gaussian convolution
division
slicing
w i w
D O W N S A M P L E
U P S A M P L E
“A Fast Approximation of the Bilateral Filter using a Signal Processing Approach”
• 3-D histogram;
• Splatting (spatial tonal space in high dim);
• Blurring (convolution); – 3D FFT.
• Division & Slicing (interpolation).
[Pham, ECCV’06]
Bilateral Grid
Bilateral Grid (importance sampling) •Bilateral grid = 3D array
–x and y correspond to pixel position
–z corresponds to pixel intensity
–Euclidean distance accounts
for edges
• space distance (x, y) and
intensity distance (z)
•Grid can be coarsely sampled; • 70 x 70 x 10 for an 8 MB pixel image
•Bilateral filter = high dim. kernel + … • Gaussian convolut. applied to grid;
• Sampling and normalization;
• Downsample, blur, then upsample.
Standard 2D Image
x
y [Chen, Siggraph’07]
Bilateral Filter on the Bilateral Grid
space
inte
nsi
ty
Image scanline
Filtered scanline
Slice: query grid with input image
Bilateral Grid
Gaussian blur grid values
space
inte
nsi
ty
Bilateral Grid for Color Image
Tiiiiii
i
i
i
bgryxp
ip
iv
iv
pixel ofposition :
pixel of ueoutput val :ˆ
pixel of ueintput val :
Image
High-Dimensional Grid (5d grid)
High Memory Usage Cost
Bilateral Filtering Based on Luminance Bilateral Filtering Based on Color
Gaussian KD-Tree
x
y
r,g,b
Input Image
Gaussian KD Tree
High-Dimensional Space
Resolution Reduction
•Monte-Carlo Sampling •Weighted Importance Sampling
Different Weighting to Leaf Nodes
Inner Node
Leaf Node
[Adams, Siggraph’09]
• Low Computational Complexity
• Low Memory Usage
• Building The Tree • Querying The Tree
• Gaussian KD-tree is also Important Sampling • High-dimension Gaussian filter: sampling s neighborhood
• Important sampling with Gaussian KD-tree: evaluating samples as near as possible s
• High dimension kernel + sampling on leaf nodes • Splatting (downsample points to leaf nodes)
• Blurring (Gaussian blurring on leaf nodes)
• Slicing (upsample from leaf nodes to points)
Difference of Gaussian Kd-Tree, Bilateral grid
Naïve Bilateral Filtering
5-D Bilateral Grid 3-D Bilateral Grid
KD-Tree
• Patch-based (not pixel-based like bilateral filter);
• Averaging with nearby pixels of similar texture;
– Pixel-wised
– Patch-wised
• More discriminative, still blurry in finer details.
• Note: Intel OpenCV implements NLM for denoising.
Nonlocal Means Filter [Buadess’05]
Non-Local Means by Gaussian Kd-Tree
Gaussian KD Tree
Inner Node
Leaf Node
Image Pixel
Different Weighting to Leaf Nodes
High-Dimensional Space
Querying
Permutohedral Lattice for High Dim • A framework for high-dim Gaussian filtering (Gauss KD-tree)
– Embedding the input values at positions in a high-dimensional space (splatting): bary centric coord. interpolation;
– Performing a Gaussian blur in that space (blurring);
– Sampling the space at the original positions (slicing).
• It tessellates the high-dimensional space with uniform simplices (HD tetrahedra);
• Less sparse than Gaussian kd-tree, but less memory consumption than bilateral grid, efficient for a high dim filter.
[Adams, Eurographic’10]
Permutohedral Lattice for High Dim
Speed Comparison of Permutohedral Lattice with Bilateral Grid and Gaussian Kd-tree
Gauss Transform and its fast version • Discrete Gauss transform
– v’(x) = Σ’y v(y) f(p(x) - p(y)); – Take a high-dimensional signal; – Put it into a data structure; – Perform a Gaussian blur really fast; – Read out its values.
• Fast Gauss Transform [Greengard &Strain’91]: reduces computation from O(MN) to O(M + N) for N Gaussians at M points in d dimensions; – The more general “Fast Multipole Method (FMM)”.
• Improved by [Yang’03]: for high dimension d, use a multivariate Taylor expansion scheme combined with the space subdivison using the k-center algorithm; – Employ “Farthest-Point Clustering”.
Fast Gauss Bilateral Filtering • Bilateral filtering as convolution: a weighted
sum of Gaussians;
• Bilateral filtering as Gauss Transform: a ratio of two (n+1)-d Gauss Transform;
• Dimensional elevation [Paris’06]: clustering and Taylor series expansion;
• Fast Gauss Transform: a fast multi-pole method strategy, O(N).
[Yoshizawa’10]
Comparison of Fast Edge Preserving Filters
Constant Time Bilateral Filtering O(1)
• [Porikli’08] • Use of integral histogram for box spatial kernel; • Use of linear filters of image powers for polynomial range kernel; • Linear filter decompositions by Taylor series for Gaussian range kernel .
Real-time Bilateral Filtering O(1) • [Yang et al.’09] • Bilateral filter can be decomposed into a number of O(1) spatial filters for
each pixel value (quantization is moved into a range function); • Output as linear interpolation of each closest principle bilateral filtered
image component pair ( I(x) = k, k+1 ); – Box filtering ->integral images; – Gaussian filtering -> recursive filtering; – Polynomial filtering -> a set of integral images; – Median filtering -> integral image.
Fast Median and Bilateral Filtering • Median filter:
– Huang’s O(r): adjacent sliding windows overlap to some extent; • Makes use of this sequential overlap to consolidate the redundant
calculations, reducing the computational complexity to O(r)
– Distributive Histograms O(log(r)): If multiple columns are processed at once, the redundant calculations become sequential. • Histogram distributivity: for disjoint image regions A and B, their
union’s window W compute its histogram by HW=HA + HB; • The union window’s median comes from scanning HW, splicing
together from HA and HB on the fly.
– High depth median filter: storage consideration • Ordinal transform: reduce the histogram size; • Compound histogram: Hc
• Coarse to fine recursion
[Weiss, Siggraph’06]
Fast Median and Bilateral Filtering
O(r) O(logr)
Fast Median and Bilateral Filtering • Bilateral filter:
– box spatial kernel: weighting function becomes constant – 3-d histogram: – logarithmic bilateral:
[Weiss, Siggraph’06]
Smoothed Local Histogram Filter
• Image filter from local histogram: – Median, Dilation/Erosion filter: 50% ,100%/0% point of the local histogram;
• How to accelerate local histogram? – [Huang75] O(n), [Weiss06] O(log(n)), [Porikli05] O(1); – Center-weighted isotropic histogram neighborhood;
• KDE (Parzen window) as a smoothed locally weighted histogram
– K determines the frequency content of as a function of s; – W determines the spatial frequency content; – For W arbitrary kernel, the convolution can performed at the cost of O (log(n))
operations per output pixel using 2D FFT (for n neighborhood size) ; – If K, W are both Gaussian, the convolution can be done in O(1) constant time,
independent of neighborhood size per pixel.
• Designed new filters: – Closest mode filter: restricted mean shift; – Dominant mode filter: choose the mode corresponding to the largest population.
)(ˆ sf p
[Kass, ACM T-Graphics’10]
Smoothed Local Histogram Filter
Accelerating Spatially Varying Gaussian Filters • Drawback of Bilateral Filter
– piecewise-flat regions, false edges, blooming at high contrast regions; – spurious detail at color edges;
• Trilateral filter: Besides, the filter window is “tilted” by the gradient vector; • Acceleration schemes of S. V. (spatially varying) filters: importance sampling
– Directly evaluate anisotropic filter kernel by integration for Gaussian kd-tree; – Embeds data in higher dim space to make kernels separable, i.e. dimensional
elevation M:(x,y) (x, y, x+y, x-y); – Segments dataset into overlapping regions of constant anisotropic kernel to
accelerate in the permutohedral lattice; – Kernel sampling sparsely and segment by clustering, then interpolate result finally.
[Baek, Siggraph Asia’10]
noisy signal
bilateraled
kernels along the gradient
l
x
y
2D Gaussian kernel 3D Gaussian kernel ?
Spatially varying Gaussian filtered
The trilateral filter is a “gradient-preserving” filter
Comparison of Bilateral and Trilateral
Bilateral filter
Trilateral filter
Comparison of Bilateral and Trilateral Bilateral
Kernel sampling
Trilateral
Blooming & false edge
Kernel sampling
Domain Transform [Gastal, Siggraph’11] • Isometry (distance preserving) between curves on 2d manifold and real lines; • 1D filtering on transformed multiple-dimension domain e.g. a case of 2D
filter on a color image (XY, RGB) • Real-time high-quality edge aware filtering of image/video based on a
dimensionality reduction
Smoothing on transformed domain Ωw ≡ Edge-preserving on Ω
Ωw is ct(u)
Domain Transform
Domain Transform [Gastal, Siggraph’11] • Implementation for image filtering with iterations
– Horizontal pass along each image row
– Vertical pass along each image column
• Normalized Convolution (NC)
• Interpolated Convolution (IC)
• Recursive Filtering (RF)
)()ˆ( pctpt
d ↗ ⇨ ad 0 .. stop propagation and preserve edge
Halve σH at each iteration. Stop while σH is small enough )/2exp( Ha
Filter Quality Comparison with Bilateral Filter, WLS, Anisotropic Diffusion and Edge-Avoid Wavelet
Guided Image Filtering [He et al.’10]
• Problems in bilateral filter – Complexity – Gradient distortion: preserve edge, not gradient;
• Guided filter: in all local windows , compute the linear coefficients and the average of in all local windows that covers pixel
• Gradient preserving: q has an edge only if I has an edge • Integral images
– O(1) time – Non-approximate
Naturally O(N) time independent of the window radius
mean
variance
Guided Image Filtering [He et al.‘10]
Linear regression
Bilateral/joint bilateral filter does not have this linear model!
Noise/texture
Patch Match [Barnes’09]
• Nearest neighbor field: nearest neighbor over all possible patch candidates for some distance function between two patches;
• Naive brute force search is expensive: O(mM^2);
• Dimensionality reduction is a limited solution;
• Popular methods: KD-trees, Locality Sensitive Hashing (LSH); – These methods can be very accurate, but are relatively slow;
• Patch Match is an iterative randomized algorithm: iteratively improve the nearest neighbor field, based on local propagation and mostly local search, until convergence; – Heavily relies on the fact that images are generally coherent;
– Search space: patch offsets vs. patches;
– Neighboring pixels have coherent matches;
– Large number of random sampling will yield some good guesses.
Patch Match [Barnes’09] • Each iteration, for each pixel:
– Randomly located patches (with the prob. that drops exp. with distance from current match);
– Check if the offsets from neighboring patches give a better matching patch; If so, adopt neighbor’s patch offset;
– Search for better patch offsets within a concentric radius around the current offset; search radius halved each time till 1 (started with the image size).
Generalized Patch Match [Barnes’10] • Dealing with higher dimensional spaces;
• Scales and Rotations: from (x, y) to (x, y, θ, s);
• Any distance function (pairwise comparisons) e.g. – SIFT, a popular computer vision descriptor;
– Patches invariant to lighting changes;
• k-Nearest Neighbors: heap-based. – Store k correspondences in max-heap;
– If new candidate better, pop heap, push new candidate.
• Outperform KD-tree and LSH by two orders of magnitude.
Ground truth Gaussian noise NLM PM Generalized PM
Coherence Sensitive Hashing [Korman & Avidan ’11]
• CSH extend LSH and Patch Match: – LSH relies on hashing by mapping similar patches to the same bin;
– PM relies on images are coherent, to propagate good matches to neighbors;
– CSH relies on hashing to seed the initial patch matching and on image coherence to propagate good matches (3~4 times faster than PM).
• Indexing and Search: Input: images A, B, Output: nearest patch map ANNF
Coherence Sensitive Hashing [Korman & Avidan ’11]
• Hash all patches of both images (with hash function g) – Keep k (=2) representatives of each image in each entry
p
Image ‘A’ Image ‘B’ k k
Hash Table
g
g
g-1 g-1
CSH Indexing
Coherence Sensitive Hashing [Korman & Avidan ’11]
Type 1 Type 2
Type 3
Type 1 Type 2 Type 3
Uses appearance
Uses coherence
# candidates 2 4 - 6 2
Estimated contribution
20% 50% 30%
(LSH) (> PM)
CSH Indexing – Candidate (potential matching patches) Types
appearance only
appearance and coherence appearance only
Coherence Sensitive Hashing [Korman & Avidan ’11]
CSH Recap Preprocessing: • Project A’s and B’s patches on Walsh-Hadamard vectors
Iterate (typically 1-3 times): • Hash patches into a table
– Recall the hash function;
– hi(v) is a binned projection of v on one of WH kernels;
– Heuristics adapt to these kernels in:
• number of bins (per projection);
• spacing of bins.
• For each patch in A
– Create candidate patches in B
– Pick the best one
A B
)()()( 1 vhvhvg n
r
bvavh ii
i )(
Propagation-Assisted KD-Tree [He & Sun ’12]
• Jointly exploit distrib. of candidates and dependency of queries;
• Build a kd-tree using all the patches in the Candidate Set. Each leaf contains no more than m candidates.
• Scan the image A in raster order. For each patch pA(x, y) in A, do propagation-assisted kd-tree search as follows: – Descend the tree to a leaf (Leaf #0);
– Propagate a leaf from left/upper, using the already matched result of pA(x-1, y) and pA(x, y-1);
– Find the NN of pA(x, y) in all the above leaves.
• Check for each query its own leaf and extra leaf propagated from nearby queries after organizing the candidates in a KD-tree;
• Check fewer candidates and has no backtracking;
• 10~20 times than PM, 2~5 times than CSH.
Propagation-Assisted KD-Tree [He & Sun ’12]
The joint distribution P(e0, e1) in different search strategies
Propagation in Patch Match Propagation-assisted KD-tree
References • Tomasi, C., Manduchi, R. “Bilateral filtering for gray and color images”. ICCV’98, pp839–
846. • F. Durand and J. Dorsey, “Fast bilateral filtering for the display of high-dynamic-range
images,” ACM T-Graphics, 21(3), pp. 257–266, (SIGGRAPH 2002). • Choudhury, P., Tumblin, J., "The Trilateral Filter for High Contrast Images and Meshes,"
Eurographics Symp. on Rendering, pp. 186-196, 2003 • T. Pham, L. Vliet, “Separable bilateral filtering for fast video preprocessing,” IEEE ICME,
2005 • S. Paris and F. Durand, “A fast approximation of the bilateral filter using a signal
processing approach,” ECCV, 2006. • B. Weiss, “Fast median and bilateral filtering,” ACM T-Graphics, 25(3), ACM SIGGRAPH,
2006. • Porikli, F. “Constant time o(1) bilateral filtering”, IEEE CVPR’08. • Q Yang, K Tan and N Ahuja, “Real-time O(1) Bilateral Filtering”, IEEE CVPR 2009. • J. Chen, S. Paris, and F. Durand, “Real-time edge-aware image processing with the
bilateral grid,” ACM Transactions on Graphics (SIGGRAPH 07). • A Adams, N. Gelfand, J. Dolson, and M. Levoy, “Gaussian KD-Trees for Fast High-
Dimensional Filtering,” ACAdams M Transactions on Graphics (SIGGRAPH 2009).
References • S Yoshizawa, A Belyaev, H Yokota, Fast Gauss Bilateral Filtering , Computer Graphics
forum’10. • M Kass, J Solomon, Smoothed Local Histogram Filters, ACM T-Graphics, 2010. • A Adams, J Baek, A Davis, “Fast High-Dimensional Filtering Using the Permutohedral
Lattice”, Eurographics 2010. • Baek, J., Jacobs, D. E. “Accelerating Spatially Varying Gaussian Filters”. SIGGRAPH
Asia 2010. • E Gastal and M. Oliveira, Domain Transform for Edge-Aware Image and Video
Processing, SIGGRAPH 2011.
• C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. “Patchmatch: a randomized correspondence algorithm for structural image editing”. SIGGRAPH, 2009.
• C. Barnes, E. Shechtman, D. B. Goldman, and A. Finkelstein. “The generalized patchmatch correspondence algorithm”. ECCV, pages 29–43, 2010.
• Kaiming He, Jian Sun, and Xiaoou Tang, “Guided image filtering”, ECCV 2010.
• S. Korman and S. Avidan. “Coherency sensitive hashing”. ICCV, 2011.
• K He, J Sun, “Computing Nearest-Neighbor Fields via Propagation-Assisted KD-Trees”, CVPR, 2012.
Thanks!