Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs...
-
Upload
bertina-fletcher -
Category
Documents
-
view
221 -
download
0
Transcript of Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs...
![Page 1: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/1.jpg)
Method of Least Squares
![Page 2: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/2.jpg)
Least Squares Method of Least Squares:
Deterministic approach
The inputs u(1), u(2), ..., u(N) are applied to the system The outputs y(1), y(2), ..., y(N) are observed
Find a model which fits the input-output relation to a (linear?) curve, f(n,u(n))
‘best’ fit by minimising the sum of the squres of the difference f - y
0 5 10 15 20 25 30 35 40 45 500
5
10
15
20
25
30
35
40
45
50
![Page 3: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/3.jpg)
Least Squares
The curve fitting problem can be formulated as
Error: Sum-of-error-squares:
Minimum (least-squares of error) is achieved when the gradient is zero
model observationsvariable
![Page 4: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/4.jpg)
Problem Statement For the inputs to the system, u(i) The observed desired response
is, d(i)
Relation is assumed to be linear
Unobservable measurement error Zero mean
White
![Page 5: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/5.jpg)
Problem Statement Design a transversal filter which finds the least squares solution
Then, sum of error squares is
![Page 6: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/6.jpg)
Data Windowing We will express the input in matrix form Depending on the limits i1 and i2 this matrix changes
Covariance Methodi1=M, i2=N
Prewindowing Methodi1=1, i2=N
Postwindowing Methodi1=M, i2=N+M1
Autocorr. Methodi1=1, i2=N+M1
![Page 7: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/7.jpg)
Error signal
Least squares (minimum of sum of squares) is achieved when
i.e., when
The minimum-error time series emin(i) is orthogonal to the time series of the input u(i-k) applied to tap k of a transversal filter of length M for k=0,1,...,M-1 when the filter is operating in its least-squares condition.
Principle of Orthogonality
!Time averaging!(For Wiener filtering)
(this was ensemble average)
![Page 8: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/8.jpg)
Corollary of Principle of Orthogonality
LS estimate of the desired response is
Multiply principle of orthogonality by wk* and take summation over k
Then
When a transversal filter operates in its least-squares condition, the least-squares estimate of the desired response -produced at the output of the filter- and the minimum estimation error time series are orthogonal to each other over time i.
![Page 9: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/9.jpg)
Energy of Minimum Error
Due to the principle of orthogonality, second and third terms are orthogonal, hence
where
, when eo(i)= 0 for all i, impossible , when the problem is underdetermined fewer data points
than parameters infinitely many solutions (no unique soln.)!
![Page 10: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/10.jpg)
Normal Equations
Hence,
Expanded system of the normal equations for linear least-squares filters.
Minimum error: Principle of Orthogonality→
(t,k), 0≤(t,k) ≤M-1time-average
autocorrelation functionof the input
z(-k), 0 ≤k ≤M-1time-average
cross-correlation bwthe desired response
and the input
![Page 11: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/11.jpg)
Normal Equations (Matrix Formulation)
Matrix form of the normal equations for linear least-squares filters:
Linear least-squares counterpart of the Wiener-Hopf eqn.s. Here and z are time averages, whereas in Wiener-Hopf eqn.s they
were ensemble averages.
(if -1 exists!)
![Page 12: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/12.jpg)
Minimum Sum of Error Squares
Energy contained in the time series is
Or,
Then the minimum sum of error squares is
![Page 13: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/13.jpg)
Properties of the Time-Average Correlation Matrix
Property I: The correlation matrix is Hermitian symmetric,
Property II: The correlation matrix is nonnegative definite,
Property III: The correlation matrix is nonsingular iff det() is nonzero
Property IV: The eigenvalues of the correlation matrix are real and non-negative.
![Page 14: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/14.jpg)
Properties of the Time-Average Correlation Matrix
Property V: The correlation matrix is the product of two rectangular Toeplitz matrices that are Hermitian transpose of each other.
![Page 15: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/15.jpg)
Normal Equations (Reformulation)
But we know that
which yields
Substituting into the minimum sum of error squares expression gives
then
! Pseudo-inverse !
![Page 16: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/16.jpg)
Projection
The LS estimate of d is given by
The matrix
is a projection operator onto the linear space spanned by the columns of data matrix A i.e. the space Ui.
The orthogonal complement projector is
![Page 17: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/17.jpg)
Projection - Example
M=2 tap filter, N=4 → N-M+1=3 Let
Then
And
orthogonal
![Page 18: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/18.jpg)
Projection - Example
![Page 19: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/19.jpg)
Uniqueness of the LS Solution
LS always has a solution, is that solution unique?
The least-squares estimate is unique if and only if the nullity (the dimension of the null space) of the data matrix A equals zero.
AKxM, (K=N-M+1)
Solution is unique when A is of full column rank, K≥M All columns of A are linearly independent Overdetermined system (more eqns. than variables (taps)) (AHA)-1 nonsingular → exists and unique
Infinitely many solutions when A has linearly dependent columns, K<M
(AHA)-1 is singular
![Page 20: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/20.jpg)
Properties of the LS Estimates
Property I: The least-squares estimate is unbiased, provided that the measurement error process eo(i) has zero mean.
Property II: When the measurement error process eo(i) is white with zero mean and variance 2, the covariance matrix of the least-squares estimate equals 2-1.
Property III: When the measurement error process eo(i) is white with zero mean, the least squares estimate is the best linear unbiased estimate.
Property IV: When the measurement error process eo(i) is white and Gaussian with zero mean, the least-squares estimate achieves the Cramer-Rao lower bound for unbiased estimates.
![Page 21: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/21.jpg)
Computation of the LS Estimates The rank (W) of an KxN (K≥N or K<N) matrix A gives
The number of linearly independent columns/rows The number of non-zero eigenvalues/singular values
The matrix is said to be full rank (full column or row rank) if
Otherwise, it is said to be rank-deficient
Rank is an important parameter for matrix inversion If K=N (square matrix) and the matrix is full rank (W=K=N) (non-
singular) inverse of the matrix can be calculated, A-1=adj(A)/det(A)
If the matrix is not square (K≠N), and/or it is rank-deficient (singular), A-1 does not exist, instead we can use the pseudo-inverse (a projection of the inverse), A+
![Page 22: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/22.jpg)
SVD We can calculate the pseudo-inverse using SVD.
Any KxN matrix (K≥N or K<N) can be decomposed using the Singular Value Decomposition (SVD) as follows:
![Page 23: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/23.jpg)
SVD
The system of eqn.s, is overdetermined if K>N, more eqn.s than unknowns,
Unique solution (if A is full-rank) Non-unique, infinitely many solutions (if A is rank-deficient)
is underdetermined if K<N, more unknowns than eqn.s, Non-unique, infinitely many solutions
In either case the solution(s) is(are)
where
![Page 24: Method of Least Squares. Least Squares Method of Least Squares: Deterministic approach The inputs u(1), u(2),..., u(N) are applied to the system The.](https://reader036.fdocuments.us/reader036/viewer/2022062422/56649ecd5503460f94bda532/html5/thumbnails/24.jpg)
Computation of the LS Estimates
Find the solution of (A: KxM)
If K>M and rank(A)=M, ( ) the unique solution is
Otherwise , infinitely many solutions, but pseudo-inverse gives the minimum-norm solution to the least squares problem.
Shortest length possible in the Euclidean norm sense.