Multi-Class Model Fitting by Energy Minimization and ... -...
Transcript of Multi-Class Model Fitting by Energy Minimization and ... -...
![Page 1: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/1.jpg)
SZTAKI 2016
Multi-Class Model Fitting by Energy Minimization and Mode-Seeking
Daniel Barath
joint work with Jiri Matas
1
![Page 2: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/2.jpg)
© SZTAKI 2015.
Multi-class Multi-instance Fitting Problem
Interpreting the input data as a set of model instances of multiple classes. 2
![Page 3: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/3.jpg)
© SZTAKI 2015.
Multi-class Multi-instance Fitting Problem
Interpreting the input data as a set of model instances of multiple classes. 3
![Page 4: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/4.jpg)
© SZTAKI 2015.
Instance of Single Class Multi Model Fitting: # Fitting multiple homographies.
4
![Page 5: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/5.jpg)
© SZTAKI 2015.
Instance of Single Class Multi Model Fitting: # Fitting multiple two-view rigid motions.
5
![Page 6: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/6.jpg)
© SZTAKI 2015.
Instance of Single Class Multi Model Fitting: # Fitting multiple motions in video sequences.
6
![Page 7: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/7.jpg)
© SZTAKI 2015.
Instance of Multi Class Multi Model Fitting: # Fitting lines and circles (or other 2D shapes) on edge map.
7
![Page 8: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/8.jpg)
© SZTAKI 2015.
Instance of Multi Class Multi Model Fitting: # Fit planes and cylinders to detect traffic signs and columns in LIDAR data
8
![Page 9: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/9.jpg)
© SZTAKI 2015.
It is and Active and Old Problem Multi-model fitting of a single class is still an open problem.
Publications from the last few years:
• D., Barath, L., Hajder, and J., Matas [BMVC 2016]
• L., Magri and A., Fisuello: [ECCV 2008, CVPR 2014, BMVC 2015, CVPR 2016]
• H. Wang, G. Xiao, Y. Yan, and D. Suter: [ICCV 2015]
• T. T. Pham, T.-J. Chin, K. Schindler, and D. Suter: [TIP 2014]
• H. Isack and Y. Boykov: [IJCV 2012]
• E. Elhamifar and R. Vidal: [CVPR 2009]
• J.-P. Tardif: [ICCV 2009]
• N. Lazic, I. Givoni, B. Frey, and P. Aarabi: [ICCV 2009]
Multi-model fitting of multiple classes???
• No recent publications in the literature.
9
![Page 10: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/10.jpg)
© SZTAKI 2015.
It is and Active and Old Problem
Multi-model fitting of multiple classes???
• No recent publications in the literature
I have two interpretations:
• Even the single-class case is barely solved: good results, but for the per-test-
tuned case. (Parameters tuned separately for each test case.)
• It becomes important in 3D and cheap 3D sensors have only been available for the
last few years.
10
![Page 11: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/11.jpg)
© SZTAKI 2015.
Energy Minimization for single class multi instance fitting PEARL: H. Isack and Y. Boykov: [IJCV 2012]
MFIGP: T. T. Pham, T.-J. Chin, K. Schindler, and D. Suter [TIP 2014]
Multi-H: D., Barath, L., Hajder, and J., Matas [BMVC 2016]
11
![Page 12: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/12.jpg)
© SZTAKI 2015.
PEARL H. Isack and Y. Boykov: [IJCV 2012]
A global energy term consisting of three terms:
1. Data term: Penalize point-to-model
assignment.
2. Spatial Regularization term: Close points
are more likely belong to the same model
instance.
3. Complexity term: Penalize the introduction of
new labels.
PEARL algorithm: iteration of labeling and
model refitting. 12
![Page 13: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/13.jpg)
© SZTAKI 2015.
MFIGP T. T. Pham, T.-J. Chin, K. Schindler, and D. Suter [TIP 2014]
Introduce geometric priors into the energy.
13
![Page 14: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/14.jpg)
© SZTAKI 2015.
Multi-H
1. Concentrating on multi-homography estimation.
2. Achieves more accurate results than state-of-the-art multi-homography estimation methods using
mode-seeking and energy minimization.
3. Doesn’t consider the general case, only homographies are fitted.
D., Barath, L., Hajder, and J., Matas [BMVC 2016]
14
![Page 15: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/15.jpg)
© SZTAKI 2015.
Multi-H D., Barath, L., Hajder, and J., Matas [BMVC 2016]
15
![Page 16: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/16.jpg)
© SZTAKI 2015. 16
Multi-X for multi class multi instance fitting
![Page 17: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/17.jpg)
© SZTAKI 2015.
Goals
1. Fit multiple model instances of different classes.
2. Having accurate results without tuning the parameters
problem-by-problem.
17
![Page 18: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/18.jpg)
© SZTAKI 2015.
Problem Formulation
18
![Page 19: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/19.jpg)
© SZTAKI 2015.
Example Model: Line Model
Line model instance:
Distance function:
Parameter vector:
Line model:
19
![Page 20: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/20.jpg)
© SZTAKI 2015.
Example Model: Circle Model
Circle model instance:
Distance function:
Parameter vector:
Circle model:
20
![Page 21: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/21.jpg)
© SZTAKI 2015.
Problem Formulation
21
Parameter vector Distance function
![Page 22: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/22.jpg)
© SZTAKI 2015.
Given: - the input data P - the multi class model H* Output: - model instances G ½H* - the labelling L assigning points from P! G minimizing an energy E.
22
Multi-class Multi-instance Fitting Problem
![Page 23: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/23.jpg)
© SZTAKI 2015.
Energy – Data Term
The term penalizing the point-to-model assignment used in the literature:
23
![Page 24: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/24.jpg)
© SZTAKI 2015.
Energy – Data Term
Assumption: randomly generated model instances form modes
around the ground truth instances in the model parameter
space.
Example:
A 2D line can be represented by a 3D vector
Represent a set of line instances in the model parameter space...
24
![Page 25: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/25.jpg)
© SZTAKI 2015.
Energy – Data Term
Line instances in their 3D space. Median-Shift, iteration #1 25
![Page 26: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/26.jpg)
© SZTAKI 2015.
Energy – Data Term
Line instances in their 3D space. Median-Shift, iteration #2 26
![Page 27: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/27.jpg)
© SZTAKI 2015.
Energy – Data Term
Line instances in their 3D space. Median-Shift, iteration #3 27
![Page 28: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/28.jpg)
© SZTAKI 2015.
Energy – Data Term
The term penalizing the point-to-mode assignment:
is a mode-seeking function.
is the mode assigned to point p.
28
![Page 29: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/29.jpg)
© SZTAKI 2015.
Energy – Spatial Coherence Term
Isack, Hossam, and Yuri Boykov. "Energy-based geometric multi-model fitting." International journal of computer vision 97.2 (2012): 123-147. 29
![Page 30: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/30.jpg)
© SZTAKI 2015.
Energy – Spatial Coherence Term
The term penalizing neighbors with different labels:
Isack, Hossam, and Yuri Boykov. "Energy-based geometric multi-model fitting." International journal of computer vision 97.2 (2012): 123-147.
Weighting parameter
Labels of point p and q
Iverson bracket
Edges in the neighborhood graph
30
![Page 31: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/31.jpg)
© SZTAKI 2015.
Energy – Complexity
The term to suppress weak model instances by penalizing the introduction of new labels.
We propose a term having different cost for each model classes:
Set of distinct labels
Penalty of class
31
![Page 32: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/32.jpg)
© SZTAKI 2015.
Overall Energy
Data term (point-to-mode assignment)
Spatial Coherence term (close points belong to the
same instance)
Regularization term (penalize new instances)
32
![Page 33: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/33.jpg)
© SZTAKI 2015.
Algorithm
33
![Page 34: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/34.jpg)
© SZTAKI 2015.
Algorithm: Input Points
Input 2D points 34
![Page 35: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/35.jpg)
© SZTAKI 2015.
Algorithm: Model Instance Generation
Generated model instances 35
![Page 36: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/36.jpg)
© SZTAKI 2015.
Algorithm: Mode-Seeking
Mode-Seeking in the line space 36
![Page 37: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/37.jpg)
© SZTAKI 2015.
Algorithm: Replacing with Mode
Replacing lines with the corresponding modes. 37
![Page 38: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/38.jpg)
© SZTAKI 2015.
Algorithm: Labeling
Label points to lines (by color) 38
![Page 39: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/39.jpg)
© SZTAKI 2015.
Algorithm: Outlier Removal
Remove outliers and instances which have not enough inliers. 39
![Page 40: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/40.jpg)
© SZTAKI 2015.
Algorithm: Instance Refitting
Refit instances w.r.t. the labeling 40
![Page 41: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/41.jpg)
© SZTAKI 2015.
Model Description and Generation
41
![Page 42: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/42.jpg)
© SZTAKI 2015.
Model Representation (2D Line Example)
Line model 1:
42
![Page 43: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/43.jpg)
© SZTAKI 2015.
Model Representation (2D Line Example)
Line model 2:
43
![Page 44: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/44.jpg)
© SZTAKI 2015.
Model Representation
44
![Page 45: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/45.jpg)
© SZTAKI 2015.
Model Representation: Two Rules
1. Represent in an orthonormal coordinate system, e.g. a
2D line by two points.
2. A minimal representation which satisfies the first
criterium.
45
![Page 46: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/46.jpg)
© SZTAKI 2015.
Model Generation
Stochastic Sampling (like RANSAC):
1. Selecting a minimal subset (MSS), e.g. 2 points for a line.
2. Fit the model to the MSS.
3. Start from 1.
46
![Page 47: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/47.jpg)
© SZTAKI 2015.
Mode-Seeking
47
![Page 48: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/48.jpg)
© SZTAKI 2015.
Mode-Seeking: Mode Types
Shapira, Lior, Shai Avidan, and Ariel Shamir. "Mode-detection via median-shift." 2009 IEEE 12th International Conference on Computer Vision. IEEE, 2009. 48
![Page 49: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/49.jpg)
© SZTAKI 2015.
Mode-Seeking: Clustering Algorithm
Clustering in arbitrary dimensions:
• K-Means is not applicable since the number of modes is
unknown.
• Mean-Shift is a good choice.
• Median-Shift is more robust than Mean-Shift. << we chose
this
Median-Shift is applied using Tukey-median. 49
![Page 50: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/50.jpg)
© SZTAKI 2015.
Mode-Seeking: Automatic Parameter Setup
Different bandwidth for all data points determined as the distance from the k-th nearest neighbor.
Georgescu, Bogdan, Ilan Shimshoni, and Peter Meer. "Mean shift based clustering in high dimensions: A texture classification example." Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on. IEEE, 2003.
50
![Page 51: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/51.jpg)
© SZTAKI 2015.
Mode-Seeking: Automatic Parameter Setup
Georgescu, Bogdan, Ilan Shimshoni, and Peter Meer. "Mean shift based clustering in high dimensions: A texture classification example." Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on. IEEE, 2003.
51
![Page 52: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/52.jpg)
© SZTAKI 2015.
3. Labeling
52
![Page 53: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/53.jpg)
© SZTAKI 2015.
Labeling
Each point is labeled to a model instance using α-expansion algorithm minimizing
energy
53
![Page 54: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/54.jpg)
© SZTAKI 2015.
Adaptive Outlier Removal
54
![Page 55: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/55.jpg)
© SZTAKI 2015.
Adaptive Outlier Removal Removal of data points too far from the assigned model.
Original labeling 55
![Page 56: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/56.jpg)
© SZTAKI 2015.
Adaptive Outlier Removal
Sorted distances Distance differences Highest difference 56
![Page 57: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/57.jpg)
© SZTAKI 2015.
Adaptive Outlier Removal
Original labeling Labeling without outliers 57
![Page 58: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/58.jpg)
© SZTAKI 2015.
Model Fitting
58
![Page 59: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/59.jpg)
© SZTAKI 2015.
Model Fitting
The task is to update the instance
parameters using the obtained labeling.
L1 model fitting using Weiszfeld
algorithm (iteratively re-weighted least-
squares).
59
![Page 60: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/60.jpg)
© SZTAKI 2015.
Convergence
60
![Page 61: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/61.jpg)
© SZTAKI 2015.
Convergence
Due to the mode-seeking the energy can increase, thus the
convergence have to be defined over the full state of the algorithm.
61
![Page 62: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/62.jpg)
© SZTAKI 2015.
Convergence 1. Mode-Seeking:
1. Instance number must decrease or hold.
2. The energy can increase.
2. Labeling:
1. Instance number does not change.
2. Energy must decrease or hold.
3. Outlier Removal:
1. Instance number does not change.
2. Energy can’t increase.
4. Model Fitting:
1. Instance number does not change.
2. Energy must decrease or hold.
Convergence is ensured since the
number of possible labelings is
finite and the model instance
number monotonically decrease.
Convergence is reached when
62
![Page 63: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/63.jpg)
© SZTAKI 2015.
Experimental Results
63
![Page 64: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/64.jpg)
© SZTAKI 2015.
Tests – Lines and Circles
64
![Page 65: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/65.jpg)
© SZTAKI 2015.
Tests – Homographies
Top row: AdelaideRMF dataset, bottom row: Multi-H dataset.
Points assigned to planes by color. 65
![Page 66: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/66.jpg)
© SZTAKI 2015.
Tests – Homographies
Misclassification error (%) for the two-view plane segmentation on AdelaideRMF test pairs.
All methods, including Multi-X, are tuned separately for each test. 66
![Page 67: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/67.jpg)
© SZTAKI 2015.
Tests – Homographies
Misclassification errors (%, average and median) for two-view plane segmentation on all the 19 pairs from AdelaideRMF test pairs using fixed parameters.
67
![Page 68: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/68.jpg)
© SZTAKI 2015.
Tests – Two-view Motions
68
![Page 69: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/69.jpg)
© SZTAKI 2015.
Tests – Two-view Motions
Misclassification errors (%) for two-view motion segmentation on the AdelaideRMF dataset.
All methods, including Multi-X, are tuned separately for each test.
69
![Page 70: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/70.jpg)
© SZTAKI 2015.
Tests – Two-view Motions
Misclassification errors (%, average and median) for two-view motion segmentation on all the 21 pairs from the AdelaideRMF dataset using fixed parameters.
70
![Page 71: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/71.jpg)
© SZTAKI 2015.
Tests – Planes and Cylinders
71
![Page 72: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/72.jpg)
© SZTAKI 2015.
Tests – Planes and Cylinders
Misclassification error (%) of simultaneous plane and cylinder fitting to LIDAR data.
All methods, including Multi-X, are tuned separately for each test.
72
![Page 73: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/73.jpg)
© SZTAKI 2015.
Tests – Motions in video sequences
73
![Page 74: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/74.jpg)
© SZTAKI 2015.
Tests – Motions in video sequences
Misclassification errors (%, average and median) for multi-motion detection on 51 videos of Hopkins dataset.
All methods, including Multi-X, are tuned separately for each test. 74
![Page 75: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/75.jpg)
© SZTAKI 2015.
Processing Time
Processing times (sec) of Multi-X (M) and T-Linkage (T) for the problem of fitting (1) lines and circles,
(2) homographies, (3) two-view motions, (4) video motions, and (5) planes and cylinders. The
number of data point is shown in the first column.
75
![Page 76: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/76.jpg)
© SZTAKI 2015.
Conclusions
1. Simultaneous fitting of models is an old open problem.
2. A novel method for the multi-class multi-instance method was proposed.
3. Energy minimization combined with mode seeking for multi model fitting
outperforms the state of the art on several problems.
4. Automatic parameter setting makes the proposed method applicable to real
world tasks without high effort on manual parameter tuning. 76
![Page 77: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/77.jpg)
© SZTAKI 2015.
Work in Progress
Multiple free-form surface (3D) and curve (2D) fitting.
A possible application: car fitting to LIDAR point cloud.
77
![Page 78: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/78.jpg)
© SZTAKI 2015.
Work in Progress
Multiple free-form surface (3D) and curve (2D) fitting.
A possible application: car fitting to LIDAR point cloud.
78
![Page 79: Multi-Class Model Fitting by Energy Minimization and ... - CMPcmp.felk.cvut.cz/cmp/events/colloquium-2017.01.19/barath-cmp_colloq... · SZTAKI 2016 Multi-Class Model Fitting by Energy](https://reader030.fdocuments.us/reader030/viewer/2022041211/5dd0dfe7d6be591ccb631f64/html5/thumbnails/79.jpg)
© SZTAKI 2015.
Thank you for your attention! Questions, please? Paper will be available on arXiv later today.
79