Cross-Based Local Multipoint Filtering

Jiangbo Lu1, Keyang Shi2, Dongbo Min1,

Liang Lin2, and Minh N. Do3

1Advanced Digital Sciences Center, 2Sun Yat-Sen University,

3Univ. of Illinois at Urbana-Champaign

Computer Vision and Pattern Recognition(CVPR), 2012.1

Outline

• Introduction

• Related Work

• Proposed Algorithm

• Experimental Results

• Conclusion

2

Introduction

3

Background

• Edge-preserving smoothing filtering:

• A key component for many computer vision applications

• Goal :

• remove noise or fine details• the structure/edge should be well preserved

• Bilateral filter(BF), Guided filter(GF)

4

Objective

• Present a cross-based framework of performing local multipoint filtering efficiently.

• Two main steps:• 1) multipoint estimation• 2) aggregation

• CLMF-0、 CLMF-1

5

Guided Filter (GF)

fixed-sized square window

Cross-Based Local Multipoint Filtering(CLMF)

adaptive window size

Related work

6

Cross-based local support decision[19]

7

[19] K. Zhang, J. Lu, and G. Lafruit. Cross-based local stereo matching using orthogonal integral images. IEEE Trans. CSVT, 19(7):1073–1079, July 2009.

Bilateral Filter[15]

8

[15] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In Proc. of ICCV, 1998.

Guided Filter[6]

9

[6] K. He, J. Sun, and X. Tang. Guided image filtering. In Proc. of ECCV, 2010.

Guided Filter[6]

10

Guided Filter[6]

•

11

ProposedAlgorithm

12

Definition

• Z : Filter input• I : Guidance image• Y : Filter output

• : estimation point

• : observation point(support pixel)

• Ωp : local support region of

• Wp : square window of a radius r

• {hp, hp, hp, hp } : cross skeleton13

0 1 2 3

Y

Z

Yi = Zi - ni

Yi = aIi + b

[19] :

If , =1

Otherwise, =0

Adaptive Scale Selection

• Decide for each direction an appropriate arm length

• Cross-based method[19]

• Running average of the intensity of all the pixels covered by the current span h

• More robust against the measurement noise

14p h span h (right arm)

Adaptive Scale Selection

• Gradient reversal artifact

15

Generalization of Local Multipoint Filtering

• Zero-order (order m = 0 ) or first-order polynomial(m=1) model:

•

•

• The model should be biased toward low-order polynomials to avoid over-fitting and gradient increase.

16

Generalization of Local Multipoint Filtering

• Zero-order (order m = 0 ) or first-order polynomial(m=1) model:

• Use “least squares” to fit the data (Similar with GF) :

• ϵ is a regularization parameter to discourage the choices of large (i≥1)17

Generalization of Local Multipoint Filtering

• Zero-order (order m = 0 ) or first-order polynomial(m=1) model:

• Solutions:

18

m=0

m=1

: the number of pixels in

: mean of I in

: variance of I in 2

• Guided Filter(GF) : multipoint estimates are averaged together

• CLMF : weighted averaged

Generalization of Local Multipoint Filtering

19

Summary & Comparison

• O(1) time linear regression and aggregation (independent of the window radius r)

20

ExperimentalResults

21

Implementation• Raw matching cost[9]:

• Winner-Take-All / Occlusion detection and filling[14]

• r = 17, R = 3, τ = 20, and τs = 20

22[9] X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang. On building an accurate stereo matching system on graphics hardware. In Proc. of GPUCV, 2011.[14] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In Proc. of CVPR, 2011.

1.Scanline filling : the lowest disparity of the spatially closest nonoccluded pixel2.Median filter :

23

CLMF-1 CLMF-0Ground Truth

Experimental Results• Middlebury evaluation

24

Tsukuba

Rank:23

Conclusion

25

Conclusion

• Propose a generic framework of performing cross-based local multipoint filtering efficiently

• CLMF-0 and CLMF-1 find very competitive applications into many computer vision

• More generalized than GF

• Cross-based technique is very friendly for GPUs[20]

• Plan to map the filters onto GPUs for speedup

26

[20] K. Zhang, J. Lu, Q. Yang, G. Lafruit, R. Lauwereins, and L. V. Gool. Real-time and accurate stereo: A scalable approach with bitwise fast voting on CUDA. IEEE Trans. CSVT, 21(7):867–878, July 2011.

Full-Image Guided Filtering for Fast Stereo Matching

Qingqing Yang, Dongxiao Li, Member, IEEE,

Lianghao Wang, and Ming Zhang

IEEE SIGNAL PROCESSING LETTERS, VOL. 20, NO. 3, MARCH 201327

Outline

• Objective

• Proposed Algorithm

• Experimental Results

• Conclusion

28

Objective

• Propose a novel full-image guided filtering method

• A novel scheme called weight propagation is proposed to compute support weights.

• Edge-preserving

• Low-complexity

29

ProposedAlgorithm

30

Filter Modeling• C : filter input

• C’i : filter output at pixel i

• Wi.j : weight of pixel pair (i,j)

• Ni : normalizing constant

31

(p.q)Pi,j : adjacent nodes on the path Pi,j Tp.q(I) : propagation function

Ω : smoothness parameterBest path : minimum propagation weight → high complexity

Choose horizontal first policy

Implementation

• Two pass model• 1)Horizontal direction in separate rows• 2)the same way in separate columns

32Pr

Implementation

• Horizontal:

• For an element r in a row, the intermediate sum of weighted value :

33

Pr

u- : the left neighbor of uu+ : the right neighbor of u

Can be further accelerated by using the two-pass scan paradigm[15]The intermediate results are stored in two temporary arrays.

Implementation

• Horizontal:

• The scan process is a sequential computation of weighted cumulative sum:

• Simply computed by:

34

Pr

horizontal path → vertical pathreduce the complexity : 4 multiplication and 8 additions (each element)

AL : the weighted cumulative sums calculated from the left to right

Implementation

• Cost Volume C:

• CBT : BT measure[18]

• CGD : absolute difference of gradient

• Winner-Take-All:

• Post-processing:• Cross checking : occlusions / mismatch pixels are filled by the

lowest disparity value of the nearest non-occluded pixel• Weighted median filter

35

[18] S. Birchfield and C. Tomasi, “A pixel dissimilarity measure that is insensitive to image sampling,” IEEE Trans. Patt. Anal. Mach. Intell., vol. 20, no. 4, pp. 401–406, 1998.

Comparison• Employ as many related pixels as possible

• Important for cost filtering in large textureless regions

36

Bilateral filter

Proposed

Comparison

37

Bilateral filter

Proposed

ExperimentalResults

38

Experimental Results

• Core Duo 3.16 GHz CPU

• 2 GB 800MHz RAM

• No parallelism technique is utilized.

• The average runtime for cost-volume filtering : 68 ms (on the Middlebury benchmark data sets)

• 27 faster than the approach [13] using guided image filtering (1850 ms).

39

[13] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz. Fast cost-volume filtering for visual correspondence and beyond. In Proc. of CVPR, 2011.

40

Experimental Results

41

Experimental Results

42

Conclusion

43

Conclusion

• The novel weight propagation method ensures support elements are assigned.

• All elements in the input signal contribute to the filtering approach.

• Outperforms all local methods on the Middlebury benchmark in terms of both speed and accuracy

44