Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit!...
Transcript of Least Squares and Data Fitting - University Of Illinoisย ยท Data fitting -not always a line fit!...
Least Squares and Data Fitting
How do we best fit a set of data points?
Data fitting
Given๐ data points { ๐ก!, ๐ฆ! , โฆ , ๐ก", ๐ฆ" }, we want to find the function ๐ฆ = ๐ฅ! + ๐ฅ" ๐ก
that best fit the data (or better, we want to find the coefficients ๐ฅ#, ๐ฅ!).
Thinking geometrically, we can think โwhat is the line that most nearly passes through all the points?โ
Linear Least Squares 1) Fitting with a line
Given๐ data points { ๐ก!, ๐ฆ! , โฆ , ๐ก", ๐ฆ" }, we want to find ๐ฅ# and ๐ฅ!such that
๐ฆ$ = ๐ฅ# + ๐ฅ! ๐ก$ โ๐ โ 1,๐
or in matrix form: Note that this system of linear equations has more equations than unknowns โOVERDETERMINED SYSTEMS๐ร๐ ๐ร๐ ๐ร๐
1 ๐ก!โฎ โฎ1 ๐ก"
๐ฅ#๐ฅ! =
๐ฆ!โฎ๐ฆ"
๐จ ๐ = ๐
We want to find the appropriate linear combination of the columns of ๐จthat makes up the vector ๐.
If a solution exists that satisfies ๐จ ๐ = ๐ then ๐ โ ๐๐๐๐๐(๐จ)
Linear Least Squaresโข In most cases, ๐ โ ๐๐๐๐๐(๐จ) and ๐จ ๐ = ๐ does not have an
exact solution!
โข Therefore, an overdetermined system is better expressed as
๐จ ๐ โ ๐
Linear Least Squaresโข Least Squares: find the solution ๐ that minimizes the residual
๐ = ๐ โ ๐จ ๐
โข Letโs define the function ๐ as the square of the 2-norm of the residual
๐ ๐ = ๐ โ ๐จ ๐ %%
Linear Least Squaresโข Least Squares: find the solution ๐ that minimizes the residual
๐ = ๐ โ ๐จ ๐
โข Letโs define the function ๐ as the square of the 2-norm of the residual
๐ ๐ = ๐ โ ๐จ ๐ %%
โข Then the least squares problem becomesmin๐๐ (๐)
โข Suppose ๐:โ" โ โ is a smooth function, then ๐ ๐ reaches a (local) maximum or minimum at a point ๐โ โ โ" only if
โ๐ ๐โ = 0
How to find the minimizer?โข To minimize the 2-norm of the residual vector
min๐๐ ๐ = ๐ โ ๐จ ๐ %
%
๐ ๐ = (๐ โ ๐จ ๐)((๐ โ ๐จ ๐)
โ๐ ๐ = 2(๐จ( ๐ โ ๐จ(๐จ ๐)
First order necessary condition:โ๐ ๐ = 0 โ ๐จ( ๐ โ ๐จ(๐จ ๐ = ๐ โ ๐จ(๐จ ๐ = ๐จ( ๐
Second order sufficient condition:๐ท%๐ ๐ = 2๐จ(๐จ2๐จ(๐จ is a positive semi-definite matrix โ the solution is a minimum
Normal Equations โ solve a linear system of equations
Linear Least Squares (another approach)โข Find ๐ = ๐จ ๐ which is closest to the vector ๐โข What is the vector ๐ = ๐จ ๐ โ ๐๐๐๐๐(๐จ) that is closest to vector ๐ in
the Euclidean norm?
When ๐ = ๐ โ ๐ = ๐ โ ๐จ ๐ is orthogonal to all columns of ๐จ, then ๐ is closest to ๐
๐จ๐ป๐ = ๐จ๐ป(๐ โ ๐จ ๐)=0 ๐จ"๐จ ๐ = ๐จ" ๐
Summary:โข ๐จ is a ๐ร๐ matrix, where ๐ > ๐. โข ๐ is the number of data pair points. ๐ is the number of parameters of the
โbest fitโ function.
โข Linear Least Squares problem ๐จ ๐ โ ๐ always has solution.
โข The Linear Least Squares solution ๐ minimizes the square of the 2-norm of the residual:
min๐
๐ โ ๐จ ๐ %%
โข One method to solve the minimization problem is to solve the system of Normal Equations
๐จ(๐จ ๐ = ๐จ( ๐
โข Letโs see some examples and discuss the limitations of this method.
Example:
Solve: ๐จ#๐จ ๐ = ๐จ# ๐
โข Does not need to be a line! For example, here we are fitting the data using a quadratic curve.
Data fitting - not always a line fit!
Linear Least Squares: The problem is linear in its coefficients!
Another examplesWe want to find the coefficients of the quadratic function that best fits the data points:
We would not want our โfitโ curve to pass through the data points exactly as we are looking to model the general trend and not capture the noise.
๐ฆ = ๐ฅ! + ๐ฅ" ๐ก + ๐ฅ# ๐ก#
Data fitting
1 ๐ก! ๐ก!"โฎ โฎ โฎ1 ๐ก# ๐ก#"
๐ฅ$๐ฅ!๐ฅ"
=๐ฆ!โฎ๐ฆ#
(๐ก$,๐ฆ$)
Solve: ๐จ#๐จ ๐ = ๐จ# ๐
Which function is not suitable for linear least squares?
A) ๐ฆ = ๐ + ๐ ๐ฅ + ๐ ๐ฅ# + ๐ ๐ฅ%B) ๐ฆ = ๐ฅ ๐ + ๐ ๐ฅ + ๐ ๐ฅ# + ๐ ๐ฅ%C) ๐ฆ = ๐ sin ๐ฅ + ๐/ cos ๐ฅD) ๐ฆ = ๐ sin ๐ฅ + ๐ฅ/ cos ๐๐ฅE) ๐ฆ = ๐ ๐&#' + ๐ ๐#'
Computational Cost๐จ#๐จ ๐ = ๐จ# ๐
โข Compute ๐จ(๐จ: ๐ ๐๐%
โข Factorize ๐จ(๐จ: LU โ ๐ %2๐
2 , Cholesky โ๐ !2๐
2
โข Solve ๐ ๐%โข Since ๐ > ๐ the overall cost is ๐ ๐๐%
Short questionsGiven the data in the table below, which of the plots shows the line of best fit in terms of least squares?
A) B)
C) D)
Short questionsGiven the data in the table below, and the least squares model
๐ฆ = ๐! + ๐% sin ๐ก๐ + ๐2 sin ๐ก๐/2 + ๐3 sin ๐ก๐/4
written in matrix form as
determine the entry ๐ด%2 of the matrix ๐จ. Note that indices start with 1.
A) โ1.0B) 1.0C) โ 0.7D) 0.7E) 0.0
Solving Linear Least Squares with SVD
๐จ is a ๐ร๐ matrix where ๐ > ๐(more points to fit than coefficient to be determined)
Normal Equations: ๐จ!๐จ ๐ = ๐จ! ๐
โข The solution ๐จ ๐ โ ๐ is unique if and only if ๐๐๐๐ ๐ = ๐(๐จ is full column rank)
โข ๐๐๐๐ ๐ = ๐ โ columns of ๐จ are linearly independent โ ๐ non-zero singular values โ ๐จ! ๐จ has only positive eigenvalues โ ๐จ!๐จ is a symmetricand positive definite matrix โ ๐จ!๐จ is invertible
๐ = ๐จ!๐จ "๐๐จ! ๐
โข If ๐๐๐๐ ๐ < ๐, then ๐จ is rank-deficient, and solution of linear least squares problem is not unique.
What we have learned so farโฆ
Condition number for Normal EquationsFinding the least square solution of ๐จ ๐ โ ๐ (where ๐จ is full rank matrix) using the Normal Equations
๐จ(๐จ ๐ = ๐จ( ๐
has some advantages, since we are solving a square system of linear equations with a symmetric matrix (and hence it is possible to use decompositions such as Cholesky Factorization)
However, the normal equations tend to worsen the conditioning of the matrix.
๐๐๐๐ ๐จ(๐จ = (๐๐๐๐ ๐จ )%
How can we solve the least square problem without squaring the condition of the matrix?
SVD to solve linear least squares problems
We want to find the least square solution of ๐จ ๐ โ ๐, where ๐จ = ๐ผ ๐บ ๐ฝ๐ป
or better expressed in reduced form: ๐จ = ๐ผ5 ๐บ๐น ๐ฝ๐ป
๐จ =โฎ โฆ โฎ๐# โฆ ๐$โฎ โฆ โฎ
๐#โฑ
๐%0โฎ0
โฆ ๐ฏ#" โฆโฎ โฎ โฎโฆ ๐ฏ%" โฆ
๐จ is a๐ร๐ rectangular matrix where ๐ > ๐, and hence the SVD decomposition is given by:
Recall Reduced SVD
๐จ = ๐ผ4 ๐บ๐น ๐ฝ๐ป
๐ร๐ ๐ร๐๐ร๐
๐ร๐
๐ > ๐
SVD to solve linear least squares problems
We want to find the least square solution of ๐จ ๐ โ ๐, where ๐จ = ๐ผ$ ๐บ๐น ๐ฝ๐ป
Normal equations: ๐จ"๐จ ๐ = ๐จ" ๐ โถ ๐ผ& ๐บ๐น ๐ฝ" " ๐ผ& ๐บ๐น ๐ฝ" ๐ = ๐ผ& ๐บ๐น ๐ฝ" "๐
๐จ =โฎ โฆ โฎ๐! โฆ ๐Aโฎ โฆ โฎ
๐!โฑ
๐A
โฆ ๐ฏ!( โฆโฎ โฎ โฎโฆ ๐ฏA( โฆ
๐จ = ๐ผ4 ๐บ๐น ๐ฝ๐ป
๐ฝ ๐บ๐น๐ผ&" ๐ผ& ๐บ๐น ๐ฝ" ๐ = ๐ฝ ๐บ๐น๐ผ&"๐
๐ฝ ๐บ๐น ๐บ๐น ๐ฝ"๐ = ๐ฝ ๐บ๐น๐ผ&"๐
๐บ๐น ( ๐ฝ"๐ = ๐บ๐น๐ผ&"๐ When can we take the inverse of the singular matrix?
๐บ๐น " ๐ฝ#๐ = ๐บ๐น๐ผ$#๐
1) Full rank matrix (๐$ โ 0 โ๐):
๐ฝ#๐ = ๐บ๐น %&๐ผ$#๐๐ = ๐ฝ ๐บ๐น "'๐ผ$! ๐
Unique solution:
๐ร๐ ๐ร๐ ๐ร๐๐ร1๐ร1
rank ๐จ = ๐
2) Rank deficient matrix ( rank ๐จ = ๐ < ๐ )
Solution is not unique!!
Find solution ๐ such thatmin๐๐ ๐ = ๐ โ ๐จ ๐ "
"
and also
๐บ๐น " ๐ฝ#๐ = ๐บ๐น๐ผ$#๐
min๐
๐ ๐
2) Rank deficient matrix (continue)
Change of variables: Set ๐ฝ#๐ = ๐ and then solve ๐บ๐น ๐ = ๐ผ$#๐ for the variable ๐
๐#โฑ
๐)0
โฑ0
๐ฆ#โฎ๐ฆ)๐ฆ)*#โฎ๐ฆ%
=
๐#"๐โฎ
๐)"๐๐)*#" ๐โฎ
๐%"๐
๐ฆ+ =๐+"๐๐+
๐ = 1,2, โฆ , ๐
What do we do when ๐ > ๐? Which choice of ๐ฆ+ will minimize
๐ ๐ = ๐ฝ ๐ ๐?
๐ฆ+ = 0, ๐ = ๐ + 1,โฆ , ๐Set
We want to find the solution ๐ that satisfies ๐บ๐น " ๐ฝ#๐ = ๐บ๐น๐ผ$#๐ and also satisfies min๐
๐ ๐
Evaluate
๐ = ๐ฝ๐ =โฎ โฆ โฎ๐& โฆ ๐)โฎ โฆ โฎ
๐ฆ&๐ฆ"โฎ๐ฆ)
๐ =9*+&
)
๐ฆ* ๐๐ = 9*+&-!./
)(๐*#๐)๐*
๐๐
Solving Least Squares Problem with SVD (summary)
โข Find ๐ that satisfies min๐
๐ โ ๐จ ๐ ))
โข Find ๐ that satisfiesmin๐
๐บ๐น ๐ โ ๐ผ$#๐ ""
โข Propose ๐ that is solution of ๐บ๐น ๐ = ๐ผ5(๐
โข Evaluate: ๐ = ๐ผ5(๐
โข Set: ๐ฆ$ = hO"P", if ๐$ โ 0
0, otherwise๐ = 1,โฆ , ๐
โข Then compute ๐ = ๐ฝ ๐
Cost:
๐ ๐
๐
๐#
Cost of SVD:๐(๐ ๐#)
โข If ๐$โ 0 for โ๐ = 1,โฆ , ๐, then the solution ๐ = ๐ฝ ๐บ๐น Q!๐ผ5( ๐ is unique (and not a โchoiceโ).
โข If at least one of the singular values is zero, then the proposed solution ๐ is the one with the smallest 2-norm ( ๐ % is minimal ) that minimizes the 2-norm of the residual ๐บ๐น ๐ โ ๐ผ5(๐ %
โข Since ๐ % = ๐ฝ ๐ %= ๐ %, then the solution ๐ is also the one with the smallest 2-norm ( ๐ % is minimal ) for all possible ๐ for which ๐จ๐ โ ๐ % is minimal.
Solving Least Squares Problem with SVD (summary)
Solve ๐จ ๐ โ ๐ or ๐ผ> ๐บ๐น๐ฝ๐ป๐ โ ๐
๐ โ ๐ฝ ๐บ๐น A ๐ผ>B ๐
Solving Least Squares Problem with SVD (summary)
Consider solving the least squares problem ๐จ ๐ โ ๐, where the singular value decomposition of the matrix ๐จ = ๐ผ ๐บ ๐ฝ๐ป๐ is:
Determine ๐ โ ๐จ ๐ (
Example:
ExampleSuppose you have ๐จ = ๐ผ ๐บ ๐ฝ๐ป๐ calculated. What is the cost of solving
min๐
๐ โ ๐จ ๐ %% ?
A) ๐(๐)B) ๐( ๐%)C) ๐(๐๐)D) ๐ ๐E) ๐( ๐%)