Digital Geometry Processing

Div. of Systems Science and InformaticsLaboratory of Digital Geometry Processing

Satoshi KANAI

- Robust Estimator-

1

Introduction

Robust Estimation− Why needed?− Least Squares Problem− Least Squares and Outliers − Hough Transform− Robust M-estimation− Weighted Least Squares− Iteratively Reweighted Least Squares (IRLS) Algorithm− Summery

References

1. C.V.Stewart: “Robust Parameter Estimation in Computer Vision “, SIAM Review, 41(3), pp.513-537 (1999) (Available on Web)

2. J. Huber, Elvezio M. Ronchetti: “Robust Statistics” (Wiley Series in Probability and Statistics), Wiley, (2009).

3. 小柳義夫： 「ロバスト推定法とデータ解析」, 日本物理学会誌，34(10), pp.884-891, (1979) （CiNiiからPDFでダウンロード可）

2

Robust Estimation

Common computational problem in vision– to estimate parameters of a model from image – Ex. Fitting lines to a set of edge pixels,

Fitting 3D planar regions to laser-scanned point clouds

Key difficulties–Must fit the models to noisy image or point clouds–Initial guesses for the models must be generated automatically– Multiple occurrences of the models in one data–The data contains “outliers” (異常値) typically

(i.e. Observations that do not belong the modelbeing fitted.)

3

An outlier

Fit line

Least Squares Problem –Fit error estimation-4

L

Parameters of the model (L)

𝑐: Distance of L from the origin

𝑒

𝑒

𝑒 ∈ ∞, ∞

Available in 2D, 3D,4D, …..

Least Squares Problem –Minimizing the squared error-5

Least Squares Problem –Least Squares Solution(1)-6

By substituting Eq(6) into the line equation 𝒏 · 𝒙 𝑐 0 , 𝒏 · 𝒙 𝒎 0 shows that the optimal line must pass trough 𝒎 .

Least Squares Problem –Least Squares Solution(2)-7

M : 2x2 𝒙𝒌 ∈ ℜ3x3 𝒙𝒌 ∈ ℜ

Least Squares Problem –Least Squares Solution(3)-8

min

Note: The eigenvector n of M is generally not unique. Therefore, n which satisfies the normalized condition n 1 should be selected.

Least Squares Problem –Least Squares Solution (Summery)-9

𝜇: The smallest eigenvalue of M𝒏: The eigenvector of M for 𝜇

Least Squares Problem –Graphical Interpretations-10

L(Least square solution)

Eigenvector for the smallest eigenvalue = the direction of minimum data distribution

Eigenvector for the largest eigenvalue = the direction of maximum data distribution

Least Squares and Outliers11

Unfortunately, least squares(LS) solutions are sensitive to outliers

LS estimate is strongly influenced by the small cluster of outliers

LS approach is NOT suitablefor the case − fitting data with outliers,− multiple solutions are

expected.

Outliers

Robust M-estimation - Overview -12

𝜌 𝑒

𝑐6

1 1𝑒𝑐

𝑒 𝑐

𝑐6

𝑒 𝑐

Robust M-estimation - Typical estimators-13

estim

ator

r

estim

ator

r

Robust M-estimation - Influence Function-14

Given the estimator 𝜌 𝑒 , the influence function 𝜓 𝑒 is defined

as 𝜓 𝑒 (16)

𝜓 𝑒 gives the sensitivity of the overall estimate 𝒏, 𝑐 on data with error 𝑒.

Examples of the estimators, influence functions and weight functions 15

Objective Function Examples - LS and L1 -

Plotting the objective function 𝐸 𝒏 𝜃 , 𝑐 for 𝒏 𝜃 cos𝜃, sin𝜃

LS estimator provides a unique local minimum L1 estimator can have multiple local minima The significant influence of data with large errors

16

LS𝜌 𝑒 𝑒

L1𝜌 𝑒 𝑒

LS

L1

θ

The shape of E around the minima is “shallow”

Objective Function Examples - GM (with 𝜎 1) -17

𝜌 𝑒

The influence of a data with large error vanishes Objective function exhibits 3 strong minima (green lines) There are many weaker local minima (red lines)

Objective Function Examples - GM (with varying 𝜎) -

𝜌 𝑒

Increasing 𝜎 in the GM-estimator smooth the objective function

Fewer local minima for larger 𝜎.

As 𝜎 is increased, outliers receive greater influence

18

𝜎 2

𝜎 4

Robust M-estimation - Weight Function (1) -

It is sometimes convenient to re-express a robust estimator in a form resembling weighted least squares such as

𝐸 𝜽 ∑ 𝜌 𝑒 𝜽 → ∑ 𝑤 𝑒 𝑒 𝜽 (a)

where 𝜽 ∶ model parameter vector

The gradient of the objective function 𝐸 𝜽 ∑ 𝜌 𝑒 𝜽with respect to 𝜽 is described as

𝜽𝜽 ∑ 𝑒 ·

𝜽𝜽

∑ 𝑒 ·𝜽

· 𝑒 𝜽 ·𝜽

𝜽 (17)

If we put 𝑤 𝑒 𝑒 𝜓 𝑒 ,

𝜽

𝜽 ∑ 𝑤 𝑒 · 𝑒 𝜽 ·𝜽

𝜽 (18)

19

Robust M-estimation - Weight Function (2) -

From eq(18),

𝜽𝜽 ∑ 𝑤 𝑒 · 𝑒 𝜽 ·

𝜽𝜽 (b)

On the other hand, from eq(a) in the last slide, if we assume

𝐸 𝜽 ∑ 𝑤 𝑒 𝑒 𝜽 ,

𝜽𝜽 ∑

𝜽𝑤 𝑒 𝑒 𝜽

If we assume 𝑤 𝑒 is directly independent of 𝜽,

𝜽𝜽 ∑ 2𝑤 𝑒 · 𝑒 𝜽 ·

𝜽𝜽

∑ 𝑤 𝑒 · 𝑒 𝜽 ·𝜽

𝜽

It provides the same result as (b).

20

Robust M-estimation - Weight Function (3) -

Therefore, by taking the weight function for the error 𝑒 as

𝑤 𝑒 𝑒 𝜓 𝑒 ,

the estimation can be treated as a weighted least square

𝐸 𝜽 ∑ 𝑤 𝑒 𝑒 𝜽 → 𝑚𝑖𝑛.

21

For 𝜌 𝑒 𝑒 , the weights are constant, 𝑤 𝑒 2, and the estimation becomes LS.

Examples of the estimators, influence functions and weight functions 22

𝜌 𝑒

𝜓 𝑒

𝑤 𝑒

𝜌 𝑒 & 𝑤 𝑒 have opposite properties!

Examples of the estimators, influence functions and weight functions 23

𝜌 𝑒

𝜓 𝑒

𝑤 𝑒

Robust M-estimation - Weight Function (5) -

Back to the “robust line fitting” problem 𝐸 𝒏, 𝑐 → 𝑚𝑖𝑛, by extending the solution of slide 10, we use the weighted mean and weighted covariance matrix for M-estimation.

• 𝒎 ∑ 𝑤 𝑒 𝒙 (19)

• 𝑀 ∑ 𝑤 𝑒 𝒙 𝒎 𝒙 𝒎 (20)

where 𝑠 ≡ ∑ 𝑤 𝑒 and 𝑒 𝒏 𝒙 𝑐 .

Similar to eq(14), a necessary condition for 𝒏, 𝑐 is that𝑀𝒏 𝜇𝒏, 𝜇 : the smallest eigenvalue of 𝑀 (21)

𝑐 𝒏 , 𝒏 1 (22)

24

Robust M-estimation - Weight Function (6) -

A necessary condition for 𝒏, 𝑐 in robust estimation is that𝑀𝒏 𝜇𝒏, 𝜇 : the smallest eigenvalue of 𝑀 (21)

𝑐 𝒏 , 𝒏 1 (22)

Properties − 𝑀 typically depends on 𝒏, 𝑐 through the weights 𝑤 𝑒 , and

therefore eq(21) is a nonlinear eigenvalue problem for 𝑛.− Only for LS case, eq(21) becomes the linear eigenvalue problem.− Eq(22) implies the line must pass through the weighted mean 𝒎.

To solve this nonlinear problem, there are many ways. Iteratively reweighted least squares algorithm is often

effective.

25

Robust M-estimation - Iteratively Reweighted Least Squares (IRLS) Algorithm -

Based on the repeated update the parameters 𝒏, 𝑐from an initial guess 𝒏 , 𝑐

1. Evaluate the weights 𝑤 𝑤 𝒏 𝒙 𝑐 .2. Compute 𝒎 and 𝑀 in eqs(19) and (20).3. Set 𝒏 to be the eigenvector for the minimum eigenvalue of 𝑀 .4. Find 𝑐 by 𝑐 𝒏 𝒎.5. Reset 𝒏 ← 𝒏, 𝑐 ← 𝑐, and repeat from 1

until the update of 𝒏, 𝑐 becomes small.

Note that the updated estimate 𝒏, 𝑐 is not necessarily a local minima of eq(14), since 𝒎 and 𝑀 were computed using 𝒏 , 𝑐instead of 𝒏, 𝑐 .

The solution may depend on the initial guess 𝒏 , 𝑐

26

Robust M-estimation - Example Results -

Line fit using GM estimator 𝜌 𝑒

𝜎 std. deviation of the noise for the observations near the dominant line From two different initial guesses (green), Iterative Reweighted Least

Square algorithm were applied (black) to get the final fit lines (Blue) Left: Outliers were correctly rejected in the final fit. Right: Converged to an undesirable local minimum in the final fit.

27

Initial guess

Initial guess

The final fit The final fit

How can we avoid the dependency on the initial guess ?

outliersinliers

Robust M-estimation - Controlling Leverage - 28

A simple strategy for dealing with this problem1. Estimate the distribution of data

support along the fitted line (eg. project the weights 𝑤 from 𝒙 onto the line and blur).

2. Determine a longest contiguous region of support ( high weight interval ) along the fitted line. ( Find an interval of support without any large gaps)

3. Reduce the weights 𝑤 in the IRLS for points 𝒙 significantly outside the region of contiguous support.(eg. Set such weights to 0)

Initial guess

ReducedWeight region

ReducedWeight region

High weights are provided.

Robust M-estimation - Leverage Example (1)-29

Iteration 0 Iteration 1 Iteration 2

Iteration 3 Iteration 4 Iteration 5

Blue: Data with significant weightsGreen segment: Contiguous region of support

Robust M-estimation - Successive Estimation -30

Edge fitting problem on the edge image The first fitted segment can the be removed from the active data set,

and repeat the process. Satisfactory fitting results are obtained.

Summery of a Robust Line Estimation Algorithm

1. Initial GuessRandomly sample from the data. Set 𝒏 , 𝑐 according to the position and orientation of sampled edge pixels.

2. Iterative Fitting Use the iteratively reweighted least squares algorithm to fit the line parameters. Maintain information about the support (from the data) for the line along its length, and use it to down-weight data likely to cause leverage problems.

3. VerificationGiven a converged solution, decide whether that line has sufficient data support(e.g. consider the sum of the weights). Discard solutions with insufficient support.

4. Model SelectionFor each verified line segment, remove the edge pixels that provide significant support for it from the original data set.

Repeat steps 1 through 3.

31

Line Finder Examples32

Supplement – Choice of the scale parameter c for estimator function-

33

Est

imat

or ρ

𝜌 𝑒

𝑐6

1 1𝑒𝑐

𝑒 𝑐

𝑐6

𝑒 𝑐

Huber

Error e

Tukey

𝜌 𝑒𝑒 𝑒 𝑐

𝑐 2 𝑒 𝑐 𝑒 𝑐

cc

𝒄 𝟏. 𝟑𝟒𝟓 𝑖𝑓 variance 𝑒 1.0

𝒄 𝟒. 𝟔𝟖𝟓 𝑖𝑓 variance 𝑒 1.0

𝑠𝒎𝒆𝒅𝒊𝒂𝒏 𝒆

𝟎. 𝟔𝟕𝟒𝟓𝜌 𝑒 → 𝜌 𝑒/𝑠

𝐼𝑓 variance 𝑒 1.0

Other Robust Fitting Methods

Trimmed LS L-estimator, R-estimator LMedS (Least Median of Squares) RANSAC(Random Sampling and Consensus) RANSAC Families

-Robust: MSAC, AMLESAC, uMLESAC-Fast: Progressive RANSAC, NAPSAC, PROSAC, GASAC-Accurate: MLESAC, QDEGSAC, MAPSAC, MSAC,

34