Bilateral and Trilateral Adaptive Support Weights in...

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering

BilateralExplanation

AdaptiveSupport Weight

TrilateralFiltering

TrilateralExplanation

Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Bilateral and Trilateral Adaptive SupportWeights in Stereo Vision

Ryan Beethe

Student, Colorado School of Mines

[email protected]

April 7, 2016

1 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Overview

1 Stereo MatchingImage ExampleBlock MatchingMatching Cost

2 Bilateral FilteringBilateral ExplanationAdaptive Support Weight

3 Trilateral FilteringTrilateral ExplanationTrilateral-based ASW

4 ASW In PracticeGPU ASWASW AlternativesTiming Comparison

2 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Stereo Matching

3 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Stereo Image Example [3]

Reference Image(left)

Target Image(right)

Disparity Image(left)

4 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Stereo Image ExampleHow Disparity Image is Generated

5 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Pixel MatchingAmbiguity issues

6 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Block MatchingBlocks of Pixels Reduce Ambiguity

7 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Block Matching

Left Image Disparity Image

8 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Matching Cost, Cost Aggregation

SAD = 2

SAD = 32

SAD = 17

9 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

SAD Limitation

What is wrong with using SAD on this reference window?

10 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Bilateral Filtering

11 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Bilateral ExplanationEdge-Preserving Version on Gaussian

Gaussian kernel (regardless of pixel values)

Bilateral “kernel” at a noisy step [4]

12 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Bilateral Example

Original Image Bilaterally Smoothed Image

13 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Adaptive Support WeightIntroduction

Reference Window Target Window 1 Target Window 2

14 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Adaptive Support WeightIndividual Support Weights

Reference Window Target Window 1 Target Window 2

Individual Window Support Weights

15 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Adaptive Support WeightCombined Support Weights

Reference Window

Target Window 1

Reference WindowSupport Weight

Target Window 1Support Weight

Combined SupportWeight

16 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Adaptive Support WeightCombined Support Weights, cont.

Reference Window

Target Window 2

Reference WindowSupport Weight

Target Window 2Support Weight

Combined SupportWeight

17 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Adaptive Support WeightImproved Stereo Output

Raw BM Output ASW Output[5]

18 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Other ASW Methods

Many alternative methods for adaptive support weight havebeen proposed based on the original bilateral-based ASWimplementation. Modifications include:

• Removing spacial component of bilateral filter

• Replacing bilateral with guided filter

• Replacing bilateral filter with some other filter

• Approximating the bilateral filter

• Calculating support weight for only the reference imageFor more discussion, see [2].

19 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Bilateral ASW Limitations

Image from [5].

20 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Trilateral Filtering

21 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Trilateral ExplanationTrilateral filter, in terms of Bilateral

22 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Trilateral ExplanationTrilateral filter, in terms of Bilateral, cont.

23 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Trilateral ExplanationCut-off Term

24 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Trilateral-based ASW

OriginalWindow

Bilateralsupportweight

Additionalsupportweight

term

Combinedsupportweight

25 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Trilateral-based ASW

Notice the trilateral-based ASW (left)outpreforms bilateral-based ASW (right)

near edges:

Images adapted from [1]

26 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Trilateral Performance

Image Algorithm nonocc all disc

Semi-Global BM 3.26 3.96 12.8Tsukuba Trilateral ASW 1.65 1.96 5.90

Bilateral ASW 1.38 1.85 6.90

Semi-Global BM 1.00 1.57 11.3Venus Trilateral ASW 0.14 0.31 1.51


Semi-Global BM 6.02 12.2 16.3Teddy Trilateral ASW 6.25 11.8 15.1


Semi-Global BM 3.06 9.75 8.90Cones Trilateral ASW 2.49 8.32 7.02


27 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

ASW in Practice

28 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

GPU Adaptive Support Weight

Preliminary GPU ASWRuns as fast as 94 ms

(Quadro K6000)

ASW (from paper)[5]Takes almost 60 seconds

(CPU-only).

29 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

OpenCV Options

Block MatchingSemi-Global Block

MatchingDisparity Bilateral

Filter

Belief Propagation and Constant-Space Belief Propagation not shown

30 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Nvidia VisionWorks

Block Matching Semi-Global Block Matching

31 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Timing ComparisonTests ran on 450x375 “cones” images

Algorithm Jetson TK1 Jetson TX1 i7 + Quadro K6000

CPU BM 29ms 29ms 12msGPU BM 18ms 9.5ms 2.2msGPU DBF 64ms 21ms 10msCV SGBM 870ms 990ms 99msVX BM 13ms 5.7ms 2.3msVX SGBM 65ms 42ms 5.1msGPU ASW 8,900ms 6,800ms 94msCPU ASW 190,000ms 200,000ms 73,000ms

Source code for test programs can be found at:https://github.com/rdbeethe/gtc_tests

32 / 36

https://github.com/rdbeethe/gtc_tests

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Conclusion

ASW Pros:

• Excellent edge accuracy

• High disparity detail

• Less noise

• Parallelizes well

ASW Cons:

• Much highercomputational complexity

• No completed open-sourceGPU version

My GPU-accelerated ASW implementation can be found at:https://github.com/rdbeethe/asw

33 / 36

https://github.com/rdbeethe/asw

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

References I

Dongming Chen, Mohsen Ardabilian, and Liming Chen.A novel trilateral filter based adaptive support weightmethod for stereo matching.IEEE Transactions on Circuits and Systems for VideoTechnology, 25:730–743, 2014.

Asmaa Hosni, Michael Bleyer, and Margrit Gelautz.Secrets of adaptive support-weight for local stereomatching.Computer Vision and Image Understanding, 117:620–632,2013.

Daniel Scharstein and Richard Szeliski.A taxonomy and evaluation of dense two-frame stereocorrespondence algorithms.International Journal of Computer Vision, 2002.

34 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

References II

C. Tomasi and R. Manduchi.Bilateral filtering for gray and color images.IEEE International Conference on Computer Vision, 1998.

Kuk-Jin Yoon and In So Kweon.Adaptive support-weight approach for correspondencesearch.IEEE Transactions on Pattern Analysis and MachineIntelligence, 28:650–656, 2006.

35 / 36

ASW Stereo

Ryan Beethe

StereoMatching

Image Example

Block Matching

Matching Cost

BilateralFiltering



TrilateralFiltering


Trilateral-basedASW

ASW InPractice

GPU ASW

ASWAlternatives

TimingComparison

Conclusion

Questions?

36 / 36

Bilateral and Trilateral Adaptive Support Weights in...

Documents

Transcript of Bilateral and Trilateral Adaptive Support Weights in...