Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and...
-
Upload
lucy-cory-wiggins -
Category
Documents
-
view
213 -
download
0
Transcript of Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and...
![Page 1: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/1.jpg)
Cross-Based Local Multipoint Filtering
Jiangbo Lu1, Keyang Shi2, Dongbo Min1,
Liang Lin2, and Minh N. Do3
1Advanced Digital Sciences Center, 2Sun Yat-Sen University,
3Univ. of Illinois at Urbana-Champaign
Computer Vision and Pattern Recognition(CVPR), 2012.1
![Page 2: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/2.jpg)
Outline
• Introduction
• Related Work
• Proposed Algorithm
• Experimental Results
• Conclusion
2
![Page 3: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/3.jpg)
Introduction
3
![Page 4: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/4.jpg)
Background
• Edge-preserving smoothing filtering:
• A key component for many computer vision applications
• Goal :
• remove noise or fine details• the structure/edge should be well preserved
• Bilateral filter(BF), Guided filter(GF)
4
![Page 5: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/5.jpg)
Objective
• Present a cross-based framework of performing local multipoint filtering efficiently.
• Two main steps:• 1) multipoint estimation• 2) aggregation
• CLMF-0、 CLMF-1
5
Guided Filter (GF)
fixed-sized square window
Cross-Based Local Multipoint Filtering(CLMF)
adaptive window size
![Page 6: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/6.jpg)
Related work
6
![Page 7: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/7.jpg)
Cross-based local support decision[19]
7
[19] K. Zhang, J. Lu, and G. Lafruit. Cross-based local stereo matching using orthogonal integral images. IEEE Trans. CSVT, 19(7):1073–1079, July 2009.
![Page 8: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/8.jpg)
Bilateral Filter[15]
8
[15] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In Proc. of ICCV, 1998.
![Page 9: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/9.jpg)
Guided Filter[6]
9
[6] K. He, J. Sun, and X. Tang. Guided image filtering. In Proc. of ECCV, 2010.
![Page 10: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/10.jpg)
Guided Filter[6]
10
![Page 11: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/11.jpg)
Guided Filter[6]
•
11
![Page 12: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/12.jpg)
ProposedAlgorithm
12
![Page 13: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/13.jpg)
Definition
• Z : Filter input• I : Guidance image• Y : Filter output
• : estimation point
• : observation point(support pixel)
• Ωp : local support region of
• Wp : square window of a radius r
• {hp, hp, hp, hp } : cross skeleton13
0 1 2 3
Y
Z
Yi = Zi - ni
Yi = aIi + b
![Page 14: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/14.jpg)
[19] :
If , =1
Otherwise, =0
Adaptive Scale Selection
• Decide for each direction an appropriate arm length
• Cross-based method[19]
• Running average of the intensity of all the pixels covered by the current span h
• More robust against the measurement noise
14p h span h (right arm)
![Page 15: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/15.jpg)
Adaptive Scale Selection
• Gradient reversal artifact
15
![Page 16: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/16.jpg)
Generalization of Local Multipoint Filtering
• Zero-order (order m = 0 ) or first-order polynomial(m=1) model:
•
•
• The model should be biased toward low-order polynomials to avoid over-fitting and gradient increase.
16
![Page 17: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/17.jpg)
Generalization of Local Multipoint Filtering
• Zero-order (order m = 0 ) or first-order polynomial(m=1) model:
• Use “least squares” to fit the data (Similar with GF) :
• ϵ is a regularization parameter to discourage the choices of large (i≥1)17
![Page 18: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/18.jpg)
Generalization of Local Multipoint Filtering
• Zero-order (order m = 0 ) or first-order polynomial(m=1) model:
• Solutions:
18
m=0
m=1
: the number of pixels in
: mean of I in
: variance of I in 2
![Page 19: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/19.jpg)
• Guided Filter(GF) : multipoint estimates are averaged together
• CLMF : weighted averaged
Generalization of Local Multipoint Filtering
19
![Page 20: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/20.jpg)
Summary & Comparison
• O(1) time linear regression and aggregation (independent of the window radius r)
20
![Page 21: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/21.jpg)
ExperimentalResults
21
![Page 22: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/22.jpg)
Implementation• Raw matching cost[9]:
• Winner-Take-All / Occlusion detection and filling[14]
• r = 17, R = 3, τ = 20, and τs = 20
22[9] X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang. On building an accurate stereo matching system on graphics hardware. In Proc. of GPUCV, 2011.[14] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In Proc. of CVPR, 2011.
1.Scanline filling : the lowest disparity of the spatially closest nonoccluded pixel2.Median filter :
![Page 23: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/23.jpg)
23
CLMF-1 CLMF-0Ground Truth
![Page 24: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/24.jpg)
Experimental Results• Middlebury evaluation
24
Tsukuba
Rank:23
![Page 25: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/25.jpg)
Conclusion
25
![Page 26: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/26.jpg)
Conclusion
• Propose a generic framework of performing cross-based local multipoint filtering efficiently
• CLMF-0 and CLMF-1 find very competitive applications into many computer vision
• More generalized than GF
• Cross-based technique is very friendly for GPUs[20]
• Plan to map the filters onto GPUs for speedup
26
[20] K. Zhang, J. Lu, Q. Yang, G. Lafruit, R. Lauwereins, and L. V. Gool. Real-time and accurate stereo: A scalable approach with bitwise fast voting on CUDA. IEEE Trans. CSVT, 21(7):867–878, July 2011.
![Page 27: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/27.jpg)
Full-Image Guided Filtering for Fast Stereo Matching
Qingqing Yang, Dongxiao Li, Member, IEEE,
Lianghao Wang, and Ming Zhang
IEEE SIGNAL PROCESSING LETTERS, VOL. 20, NO. 3, MARCH 201327
![Page 28: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/28.jpg)
Outline
• Objective
• Proposed Algorithm
• Experimental Results
• Conclusion
28
![Page 29: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/29.jpg)
Objective
• Propose a novel full-image guided filtering method
• A novel scheme called weight propagation is proposed to compute support weights.
• Edge-preserving
• Low-complexity
29
![Page 30: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/30.jpg)
ProposedAlgorithm
30
![Page 31: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/31.jpg)
Filter Modeling• C : filter input
• C’i : filter output at pixel i
• Wi.j : weight of pixel pair (i,j)
• Ni : normalizing constant
31
(p.q)Pi,j : adjacent nodes on the path Pi,j Tp.q(I) : propagation function
Ω : smoothness parameterBest path : minimum propagation weight → high complexity
Choose horizontal first policy
![Page 32: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/32.jpg)
Implementation
• Two pass model• 1)Horizontal direction in separate rows• 2)the same way in separate columns
32Pr
![Page 33: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/33.jpg)
Implementation
• Horizontal:
• For an element r in a row, the intermediate sum of weighted value :
33
Pr
u- : the left neighbor of uu+ : the right neighbor of u
Can be further accelerated by using the two-pass scan paradigm[15]The intermediate results are stored in two temporary arrays.
![Page 34: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/34.jpg)
Implementation
• Horizontal:
• The scan process is a sequential computation of weighted cumulative sum:
• Simply computed by:
34
Pr
horizontal path → vertical pathreduce the complexity : 4 multiplication and 8 additions (each element)
AL : the weighted cumulative sums calculated from the left to right
![Page 35: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/35.jpg)
Implementation
• Cost Volume C:
• CBT : BT measure[18]
• CGD : absolute difference of gradient
• Winner-Take-All:
• Post-processing:• Cross checking : occlusions / mismatch pixels are filled by the
lowest disparity value of the nearest non-occluded pixel• Weighted median filter
35
[18] S. Birchfield and C. Tomasi, “A pixel dissimilarity measure that is insensitive to image sampling,” IEEE Trans. Patt. Anal. Mach. Intell., vol. 20, no. 4, pp. 401–406, 1998.
![Page 36: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/36.jpg)
Comparison• Employ as many related pixels as possible
• Important for cost filtering in large textureless regions
36
Bilateral filter
Proposed
![Page 37: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/37.jpg)
Comparison
37
Bilateral filter
Proposed
![Page 38: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/38.jpg)
ExperimentalResults
38
![Page 39: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/39.jpg)
Experimental Results
• Core Duo 3.16 GHz CPU
• 2 GB 800MHz RAM
• No parallelism technique is utilized.
• The average runtime for cost-volume filtering : 68 ms (on the Middlebury benchmark data sets)
• 27 faster than the approach [13] using guided image filtering (1850 ms).
39
[13] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In Proc. of CVPR, 2011.
![Page 40: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/40.jpg)
40
![Page 41: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/41.jpg)
Experimental Results
41
![Page 42: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/42.jpg)
Experimental Results
42
![Page 43: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/43.jpg)
Conclusion
43
![Page 44: Cross-Based Local Multipoint Filtering Jiangbo Lu 1, Keyang Shi 2, Dongbo Min 1, Liang Lin 2, and Minh N. Do 3 1 Advanced Digital Sciences Center, 2 Sun.](https://reader035.fdocuments.us/reader035/viewer/2022062718/56649eb55503460f94bbd409/html5/thumbnails/44.jpg)
Conclusion
• The novel weight propagation method ensures support elements are assigned.
• All elements in the input signal contribute to the filtering approach.
• Outperforms all local methods on the Middlebury benchmark in terms of both speed and accuracy
44