Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems...
-
date post
19-Dec-2015 -
Category
Documents
-
view
223 -
download
1
Transcript of Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems...
![Page 1: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/1.jpg)
Chapter 5Orthogonality
![Page 2: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/2.jpg)
Outline
Scalar Product in Rn
Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt Orthogonalization Process
![Page 3: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/3.jpg)
Scalar product in Rn
8
2
3
4
1
2
3
:Example
...2211
yx
yx
yxyxyxyx
T
nnT
![Page 4: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/4.jpg)
Def: Let and be vectors in either R2 or R3.
The distance between and is defined to
be the number yx
x
y
x
y
![Page 5: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/5.jpg)
Theorem 5.1.1
If and are two nonzero vectors in either R2 or
R3 and is the angle between them , then
cosTx xy y
x
y
![Page 6: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/6.jpg)
Proof: By the law of cosines,
cos2222
yxyxxy
2 2 21cos
2x y x y y x
3 3 3
22 2
1 1 1
1
2 i i i ii i i
x y y x
x y3
1i i
i
x y
![Page 7: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/7.jpg)
Corollary 5.1.2(Cauchy-Schwarz Inequality)
If and are vectors in either R2 or R3, then
With equality holding if and only if one of the
vectors is or one vector is a multiple of the
other.
Tx y x y
x
y
0
![Page 8: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/8.jpg)
&x y
cosx y
x y
1 1
x y
x y
Note: If is the angle between , then
Thus
![Page 9: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/9.jpg)
Def: The vectors and in R2(or R3)are said to
be orthogonal if .
6
4-
2
3 :B
, 0 :A
Examples2Rxx
x y
0Tx y
![Page 10: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/10.jpg)
Scalar and Vector Projections
Scalar projection of onto :
Vector projection of onto :
1
T
T
T
x y
x y
y
x y
x yp u y y
y y y
![Page 11: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/11.jpg)
x
u
y
z x p ������������� �
p u
cosx
![Page 12: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/12.jpg)
Example: Find the point
on the line that
is closest to the point
(1,4)
Sol: Note that the vector is on the line
Thus the desired point is
xy3
1
3
1w
xy3
1
2.1
0.7
v ww
w w
v
Q
xy3
1
4.1
![Page 13: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/13.jpg)
Example: Find the equation of the plane
passing through and
normal to
Sol:
4,3,2
3,1,2
0
3
1
2
4
3
2
z
y
x
0341322 zyx
![Page 14: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/14.jpg)
Example: Find the distance form
to the plane
Sol: a normal vector to
the plane is
The distance
0,0,2P
022 zyx
2
2
1
3
2 P n
n
��������������
n
P��������������
![Page 15: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/15.jpg)
Application 1: Information Retrieval Revisited Table 1
Frequency of Key words
Modules
Key Words M1 M2 M3 M4 M5 M6 M7 M8
determines 0 6 3 0 1 0 1 1
eignvalues 0 0 0 0 0 5 3 2
linear 5 4 4 5 4 0 3 3
matrices 6 5 3 3 4 4 3 2
numerical 0 0 0 0 3 0 4 3
orthogonality 0 0 0 0 4 6 0 2
spaces 0 0 5 2 3 3 0 1
systems 5 3 3 2 2 2 1 1
transformations 0 0 0 5 3 3 1 0
vector 0 4 4 3 2 2 0 3
![Page 16: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/16.jpg)
Application I: Information Retrieval Revisited
A is the matrix corresponding to Table I, then the columns of the database matrix Q are determined by setting
To do a search for the key words orthogonality, spaces, vector, we form a unit search vector whose entries are all zero except for the three rows(be put in each of the rows) corresponding to the search rows.
x
1 1, ... ,8j j
j
q a ja
1
3
![Page 17: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/17.jpg)
0.000 0.594 0.327 0.000 0.100 0.000 0.147 0.154
0.000 0.000 0.000 0.000 0.000 0.500 0.442 0.309
0.539 0.396 0.436 0.574 0.400 0.000 0.442 0.463
0.647 0.495 0.327 0.344 0.400 0.400 0.442 0.309
0.000 0.000 0.000 0.000 0.300 0.000 0.590 0.4Q
63
0.000 0.000 0.000 0.000 0.400 0.600 0.000 0.309
0.000 0.000 0.546 0.229 0.300 0.300 0.000 0.154
0.539 0.297 0.327 0.229 0.400 0.200 0.147 0.154
0.000 0.000 0.000 0.574 0.100 0.300 0.147 0.000
0.000 0.396 0.436 0.344 0.400 0.200 0.000 0.4
0.000
0.000
0.000
0.000
0.000
0.577
0.577
0.000
0.000
63 0.577
x
i
cos
where is the angle between the unit vectors and .
T Ti i i
i
y Q x y q x
x q
For our example,
0.000, 0.229, 0.567, 0.310, 0.635, 0.577, 0.000, 0.535T
y
![Page 18: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/18.jpg)
Application I: Information Retrieval Revisited
Since is the entry of that is closest to 1,this indicates that the direction of the search vector is closest to the direction of and hence that Module 5 is the one that best matches our search criteria.
y
x
5q
5 0.635y
![Page 19: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/19.jpg)
Application 2: Correlation And Covariance Matrices
Table 2
Math Scores Fall 1996
Scores
Student Assignment Exams Final
S1 198 200 196
S2 160 165 165
S3 158 158 133
S4 150 165 91
S5 175 182 151
S6 134 135 101
S7 152 136 80
Average 161 163 131
37 37 65
1 2 34
3 5 2
11 2 40
14 19 20
27 28 30
9 27 51
X
![Page 20: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/20.jpg)
The column vectors of X represent the deviations from the mean for each of the three sets of scores.
The three sets of translated data specified by the column vectors of X all have mean 0 and all sum to 0.
A cosine value near 1 indicates that the two sets of scores are highly correlated.
Scale to make them unit vectors
Application 2: Correlation And Covariance Matrices
1 2 and x x
1 1 2 21 2
1 1 and u x u x
x x
![Page 21: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/21.jpg)
If we set , thenTC U U
0.74 0.65 0.62
0.02 0.03 0.33
0.06 0.09 0.02
0.22 0.03 0.38
0.28 0.33 0.19
0.54 0.49 0.29
0.18 0.47 0.49
U
1 0.92 0.83
0.92 1 0.83
0.83 0.83 1
C
![Page 22: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/22.jpg)
The matrix C is referred to as a correlation matrix.
The three sets of scores in our example are all positively correlated since the correlation coefficients are all positive.
A negative coefficient would indicate that two data sets were negatively correlated.
A coefficient of 0 would indicate that they were uncorrelated.
Application 2: Correlation And Covariance Matrices
![Page 23: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/23.jpg)
Def: Two subspaces X and Y of are said to be orthogonal if = 0 for every and If X and Y are orthogonal, we write
x X y Y
X Y
5-2 Orthogonal Subspaces
31 2 3
Example:
Let { , } , { }
then
X span e e Y span e
X Y
Tx y
n
![Page 24: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/24.jpg)
Def: Let Y be a subspace of . The set of all vectors in that are orthogonal to every vector in Y will be denoted . Thus = { for every } The set is called the orthogonal complement of Y
Y
| 0n Tx R x y Y y Y
Y
31
2 3
Example:
Let { }
then { , }
X span e
X span e e
nn
![Page 25: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/25.jpg)
Remarks:
1. If X and Y are orthogonal subspaces of , then .
2. If Y is a subspace of , then is also a subspace of .
{0}X Y
Y
2Proof(1). If and , then 0
and hence 0.
Tx X Y X Y x x x
x
1 2
1 2 1 2
Proof(2). If and is a scalar, then for any ,
( ) ( ) 0 0
If and , then
( ) 0
T T
T TT
x Y y Y
x y x y x Y
x x Y
x x y x y x y
1 2
0 0 for each ,
.
Therefore, is a subspace of .n
y Y
x x Y
Y
nn n
![Page 26: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/26.jpg)
Four Fundamental Subspaces
Let nmA
mnAor :x AxmnA
0n nN A x Ax
It will be shown later that N A R A , N A R A
and n N A R A
ARANm
for some m n mR A b b Ax x
0T m T mN A x A x
for some T n T m nR A b b A x x
![Page 27: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/27.jpg)
Theorem 5.2.1(Fundamental Subspace Theorem)
pf: Let and
Also, if
Similarly,
x N A y R A
,: 0 1, ---------(1)A i x i m
1
and :, for some 's --------(2)m
i ii
y A i
(1)
1
:, 0m
ii
x y x A i
ARAN
N A R A
( ,:) 0, 1, 0 ( )A i z i m Az z N A
R A N A
hence, N A R A
:, 0, 1,z R A z A i i m
ARARAN
( ) (Let , then and ) ) ( .( )Tm n TN A R A N A R AA
![Page 28: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/28.jpg)
Example: Let
Clearly,
00
21
02
01AA
1
0spanAN
1
2spanAN
2
1spanAR
0
1spanAR
ARAN
ARAN
![Page 29: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/29.jpg)
Theorem 5.2.2
If S is a subspace of , then
Furthermore, if { } is a basis for S and
{ }is a basis for , then { , }
is a basis for .1,..., rx x
1,...,r nx x
S1,..., rx x
1,...,r nx x
n
n
dim dimS S n
![Page 30: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/30.jpg)
Proof: If The result follows
Suppose . Let
and
{0} nS S
{0}S
1( ) n rrX x x
( ) ( )rank X r rank X ( )R X S
5.2.1
( ) ( )Theorem
S R X N X
3.6.4
dim( ) dim ( )Thm
S N X n r
![Page 31: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/31.jpg)
To show that is a basis for ,
It remains to show their independency.
Let . Then
Similarly,
n
1
0n
i ii
c x
x S
1 1
0n r
i i i ii i
x c x x c x
1
0 0, 1r
i i ii
c x c i r
y S
1 1
0n n
i i i ii i r
y c x y c x
1
0 0, 1,n
i i ii r
c x c i r n
1{ }nx x
![Page 32: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/32.jpg)
Def: If U and V are subspaces of a vector space W
and each can be written uniquely as a
sum , where and ,then we
say that W is a direct sum of U and V, and we
write
w W
u U
v V
W U V
u v
![Page 33: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/33.jpg)
Theorem5.2.3: If is a subspace of ,
then
pf: By Theorem5.2.2,
To show uniqueness,
Suppose
where
nS SSn
SSn
, , & .nx x u v u S v S
1 1 2 2x u v u v
1 2 1 2, & ,u u S v v S
1 2 2 1u u v v S S {0}S S
1 2 2 1&u u v v
![Page 34: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/34.jpg)
Theorem5.2.4: If is a subspace of ,
then
pf: Let
If
S nSS )(
rS )dim(
rSTheorem
)dim(2.2.5
, then 0x S x y y S
x S
SS
SS
![Page 35: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/35.jpg)
Remark: Let . i.e. , Since
and
are bijections .
nmA mnA : ARANn
ArankArank )( ArankAnullityn ArankAnullity
ARARA :
ARARA :
![Page 36: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/36.jpg)
Let nmA nA :
mA :
m
n
A
A
A
A
AN
AN
AN
AN
bijection
bijection
0
0
![Page 37: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/37.jpg)
Cor5.2.5:
Let and . Then
either
(i)
or (ii)
pf:
nmA mb
nx Ax b
my
0 and 0A y y b
(i) ( ) nb R A x Ax b
(ii) ( ) ( ) ( ) 0b R A N A y N A y b
0 & 0my A y y b
( ) ( )m TR A N A )(AR
b
)( AN
for 3m
![Page 38: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/38.jpg)
Example: Let . Find
The basic idea is that the row space and the sol. of
are invariant under row operations.
Sol: (i)
(Why?)
(ii)
(Why?)
(iii) Similarly,
and
(iv) Clearly,
431
110
211
A )(),(),(),( ARANARAN
Ax b
000
110
101
~ r
row
AA
1
1
0
1
0
1
)( spanAR
1 3 2 30 0 & 0rA x x x x x
1
1
1
)( spanAN
000
210
101
~row
A
2
1
0
1
0
1
)( spanAR
1
2
1
)( spanAN
)()(& ARANARAN
![Page 39: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/39.jpg)
Example: Let
(i)
and
(ii) The mapping
and
(iv) What is the matrix representation for ?
23:0
0
3
0
0
2
A
0
1
0
0
0
1
1
0
03 spanspanARAN
2)( AR
( ): ( ) is a bijection
R AA R A R A
0
3
2
02
1
2
1
x
x
x
x
)(:1
)(
ARARAAR
03
12
1
2
1
2
1 y
y
y
y
)( ARA
![Page 40: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/40.jpg)
5-4 Inner Product Spaces
A tool to measure the
orthogonality of two vectors in
general vector space
![Page 41: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/41.jpg)
Def: An inner product on a
vector space is a function
Satisfying the following conditions:
(i) with equality iff
(ii)
(iii)
V
, : ( )V V F orC
, 0x x 0x
, ,x y y x
, , ,x y z x z y z
![Page 42: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/42.jpg)
Example: (i) Let
Then is an inner product of
(ii) Let , Then is an
inner product of (iii) Let and then
is an inner product of
(iv) Let , is a positive function and are distinct real numbers. Then
is an inner product of
, & 0 1, .nix y w i n
1
,n
i i ii
x y w x y
n
ij
n
jij
m
i
baBA
11
,nmBA ,nm ].,[)(,, 0 baCxwgf 0)( xw
b
a
dxxgxfxwgf )()()(,
].,[0 baC
, np g P )(xw
nxx 1
)()()(,1
ii
n
ii xgxPxwgp
nP
![Page 43: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/43.jpg)
Def: Let be an inner product of a
vector space and .
we say
The length or norm of is given
by
,
V ,u v V
, 0u v u v
v
,v v v
![Page 44: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/44.jpg)
Theorem5.4.1: (The Pythagorean Law)
pf:
2 2 2u v u v u v
2 2u v
2,u v u v u v
, , , ,u u u v v u v v
u
v u v
![Page 45: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/45.jpg)
Example 1: Consider with inner product
(i)
(ii)
(iii)
(iv) (Pythagorean Law)
or
]1,1[0 C
1
1)()(, dxxgxfgf
xxdxx 101,11
1
212111,11
1 dx
32
3
2,
1
1 xxdxxxx
3
8
3
2211
222 xx
3
8)1(1,11
21
1
2 dxxxxx
![Page 46: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/46.jpg)
Example 2: Consider with inner product
It can be shown that
(i)
(ii)
(iii)
Thus is an orthonormal
set.
0[ , ]C
dxxgxfgf )()(
1,
0sin,cos mxnx
cos ,cos
sin ,sin
mn
mn
mx nx
mx nx
1,cos ,sin
2nx nx n N
1 1, 1
2 2
![Page 47: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/47.jpg)
Remark
Remark: The inner product in example 2 plays a key
role in Fourier analysis application involving trigo-
nometric approximation of functions.
2 2cos sin cos sin 2x x x x
![Page 48: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/48.jpg)
Example 3: Let
and let
Then is not orthogonal to
, ,m nA B
4
0
1
3
3
1
,
3
2
1
3
1
1
BA
ABA 6, B
6,5 FF
BA
1 1 1
, ( ) ( )
,
m n mT T
ij ij iii j i
F
A B a b trace AB AB
A A A
![Page 49: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/49.jpg)
Def: Let be two vectors in an
inner product space . Then
the scalar projection of onto is
defined as
The vector projection of onto is
& 0u v
V
u
v
,1,
u vu v
vv
,1
,
u vp v v
v v v
u
v
![Page 50: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/50.jpg)
Lemma: Let be the vector projection
of onto . Then
for some
pf:
0 &v p
u v
( )
( )
i u p p
i u p u kv
2 2
( ) , , ,
, , 0
, ,
( ) .
i p u p p u p p
u v u v
v v v v
p u p
ii trivial
u
v p
u p
k
![Page 51: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/51.jpg)
Theorem5.4.2: (Cauchy-Schwarz Inequality)
Let be two vectors in an
inner product space . Then
Moreover, equality holds are linear dependent.
pf: If
If
Equality holds
i.e., equality holds iff are linear dependent.
&u v
V,u v u v
&u v
0, then , 0
0, then
v u v u v
v
2
2 2 2,
,
Pythagorean Theoremu vp u u p
v v
2 2 2 2 2 22,u v u v v u p u v
,0, or
,
u vv u p v
v v
&u v
u
v p
u p
![Page 52: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/52.jpg)
Note: From Cauchy-Schwarz Inequality for .
This, we can define as the angle between the two nonzero vectors
,1 1 if 0 and 0
,! 0, cos .
u vu v
u v
u v
u v
& .u v
F
![Page 53: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/53.jpg)
Def: Let be a vector space a function
is said to be a norm if it satisfies
V
: {0}V v v
( ) 0 with equality 0
( ) , scalar .
( ) (triangle inequality)
i v v
ii v v
iii v w v w
Remark: Such a vector space is called a normed linear space.
![Page 54: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/54.jpg)
Theorem5.4.3: If is an inner product
space, then
defines a norm on
pf: trivial
Def: The distance between is defined
as.u v
V
.V
&u v
, v v v v V
![Page 55: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/55.jpg)
Example: Let , thennx
11
1
1
1
2
21
( ) is a norm.
( ) is a norm.
( ) is a norm for any 1.
in particular =2,
, is the euclidean norm.
max
n
ii
ii n
n PP
iPi
n
ii
i x x
ii x x
iii x x p
p
x x x x
![Page 56: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/56.jpg)
Example: Let
Thus,
However,
(Why?)
1 2
1 4&
2 2x x
1 2, 0x x
2 2 2
2 1 22 2 25 20 25x x x x
2 2
1 2
2
1 2
4 16
20 16
x x
x x
Remark: In the case of a norm that is not derived from an inner product, the Pythagorean Law will not hold.
![Page 57: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/57.jpg)
Example: Let ,
then
4
5
3
x
1
2
12
5 2
5
x
x
x
![Page 58: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/58.jpg)
Example: Let
Then
2 1B x x
1B2B B
1 1 1
![Page 59: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/59.jpg)
5-3 Least Squares Problems
![Page 60: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/60.jpg)
Least squares problems
A typical example: Given
Find the best line to fit the data . or
or find such that is minimum Geometrical meaning :
, 1,i
i
xi n
y
0 1y c c x 1 1
2 20
1
1
1
1 n n
x y
x ycsolve
c
x y
Ac y
0 1,c cAc y
0 1y c c x
( , )n nx y
1 1( , )x y
![Page 61: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/61.jpg)
Least squares problems:
Given
then the equation
may not have solutions
The objective of least square problem is
trying to find such that
is minimum value
i.e., find satisfying
& ,m n mA b
Ax b
. ., ( ) ( )i e b Col A R A
x
b Ax
x
minnx
b A x b Ax
( )R A
b
Ax
![Page 62: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/62.jpg)
Preview of the results:
It will be shown that
If columns of are linear independent .
( )! ( ), min
y R Ap R A b p y b
( ) ( )
0
0
b p R A N A
A b p
A b Ax
A Ax A b
1x A A A b
A
b
( )R Ap
![Page 63: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/63.jpg)
Theorem5.3.1: Let be a subspace of , then
(i) for all
(ii)
pf:
(i)
where
If
(ii) follows directly from (i) by noting that
S m, ! , mb p S b y b p
\{ }y S p
miny S
p b y b b p S
m S S
b p z
&p S z S
\{ }y S p
2
2 2 2
0
Pythogorean Theorem
Sz S
b y b p p y b p p y
b p z S
p
b
S
unique expression
![Page 64: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/64.jpg)
Question: How to find which solves
Ans.:
From previous Theorem , we know that
x
min ?nx
A x b b Ax
( ) ( )
( ) 0
0
b p R A N A
A b p
A b A Ax
( )R A
p Ax
b
Definition : is called normal equatio .ntheA Ax A b
![Page 65: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/65.jpg)
Remark: In general, it is possible to have more than one solution to the normal equation. If is a solution, then the general solution is of the form
ˆ where ( )x h h N A
x̂
![Page 66: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/66.jpg)
Theorem5.3.2: Let and
Then the normal equation
has an unique solution .
and is the unique least squares solution to
pf: To show that is nonsingular
nmA .)( nArank A Ax A b
1x A A A b
x
.Ax b
AA
Let 0 ( ) ( ) {0}
0 0 ( ( ) )
T TA Ax Ax N A R A
Ax x rank A n
1 is the unique solution.x A A A b
![Page 67: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/67.jpg)
Note: The projection vector
is the element of that
is closet to in the least squares
sense .
Thus, The matrix is called the
projection matrix (that project any vector of
to )
1p Ax A A A A b
)(AR
b
1P A A A A
)(ARm
b
)(AR
p
![Page 68: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/68.jpg)
Suppose a spring obeys the Hook’s law
and a series of data are taken (with measurement
error) as
How to determine ?
sol: Note that is inconsistent
The normal equation is
so,
KxF
11
8
7
5
4
3
x
F
8
5
3
11
7
4
811
57
34
Kor
K
K
K
8
5
3
1174
11
7
4
1174 K
K
186 135 0.726K K
Application 2: Spring Constants
![Page 69: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/69.jpg)
Example 2: Given the data
Find the best least squares fit by a linear function.
sol: Let the desired linear function be The problem becomes to find the least squares solution of
is the unique solution.
Thus, the best linear least square fit is
5
6
4
3
1
0
y
x
xccy 10
0
1
1 0 1
1 3 4
1 6 5c
yA
c
c
10
1
4
32
3
cA A A y
c
xy3
2
3
4
∵ rank(A)=2
![Page 70: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/70.jpg)
Example3: Find the best quadratic least squares fit to the data
sol:
Let the desired quadratic function be
The problem becomes to find the least square
solution of
is the unique solution.
Thus, the best quadratic least square fit is
0
1
1
2
2.75
0.25
0.25
c
c A A A y
c
4
3
4
2
2
1
3
0
y
x
2210 xcxccy
2
1
0
9
4
1
0
3
2
1
0
1
1
1
1
4
4
2
3
c
c
c
225.025.075.2 xxy
∵ rank(A)=3
![Page 71: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/71.jpg)
5-5 Orthonormal Sets
![Page 72: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/72.jpg)
Orthonormal Set
Simplify the least squares solution
(avoid computing inverse) Numerical computational stability
![Page 73: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/73.jpg)
Def: is said to be an orthogonal set in
an inner product space if
Moreover, if , then is said
to be orthonormal.
1 nv v
V
, 0i jv v for all i j
,i j ijv v 1 nv v
![Page 74: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/74.jpg)
Example 2:
is an
orthogonal set but not orthonormal.
However ,
is orthonormal.
1 2 3
1 2 4
1 , 1 , 5
1 3 1
v v v
1 1 2 2 3 3
1 1 1, ,
3 14 42u v u v u v
![Page 75: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/75.jpg)
Theorem5.5.1: Let be an orthogonal
set of nonzero vectors in an inner product
space . Then they are linear independent.
pf: Suppose that
1 nv v
V
1
0n
i ii
c v
1 1
1
0 , , ,
0, 1,
is linearly independent.
n n
j i i i j i j j ji i
j
n
v c v c v v c v v
c j n
v v
![Page 76: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/76.jpg)
Example:
is an
orthonormal set of with inner
product .
Note: Now you know the meaning what one
says that .
1, cos , sin
2nx nx n N
,0 C
dxxgxfgf )()(
1,
xx sincos
![Page 77: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/77.jpg)
Theorem5.5.2: Let be an orthonormal
basis for an inner product space .
If , then .
pf:
1 nu u
V
1
n
i ii
v c u
,i ic u v
1 1
1
, , ,
n n
i i j j j i jj j
n
j ij ij
u v u c u c u u
c c
![Page 78: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/78.jpg)
Cor: Let be an orthonormal basis for an inner product space .
If and , then .
pf:
V
1 nu u
1
n
i ii
u a u
1
n
i ii
v b u
1
,n
i ii
u v a b
1 1
5.5.2
1
, , ,
n n
i i i ii i
nTheorem
i ii
u v a u v a u v
a b
![Page 79: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/79.jpg)
Cor: (Parseval’s Formula)
If is an orthonormal basis for an
inner product space and , then
pf: By Corollary 5.5.3,
1 nu u
V1
n
i ii
v c u
2 2
1
n
ii
v c
2 2
1
,n
ii
v v v c
![Page 80: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/80.jpg)
Example 4:
and form
an orthonormal basis for .
If , then
and
1
1
21
2
u
2
1
21
2
u
21 2
2
xx
x
1 2 1 21 2
5.5.21 2 1 2
1 2
, , ,2 2
2 2
Theorem
x x x xx u x u
x x x xx u u
2 22 2 21 2 1 2
1 22 2
x x x xx x x
![Page 81: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/81.jpg)
Example 5: Determine without computing
antiderivatives .
sol:
xdx
4sin
24 2 2 2
2
0
24 2
2 2
sin sin ,sin sin
1 cos 2 1 1 1sin cos 2
2 22 2
1and ,cos 2 is an orthonormal set of ,
2
sin sin
1 1 3
2 42
xdx x x x
xx x
x C
xdx x
![Page 82: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/82.jpg)
Def: is said to be an orthogonal matrix if the column vectors of form an
orthonormal set in .
Example 6:
The rotational matrix
and the elementary reflection matrix
are orthogonal matrix .
n nQ
nQ
cossin
sincos
cossin
sincos
![Page 83: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/83.jpg)
Properties of orthogonal matrices:
If is orthogonal, thenn nQ
1
2 2
( ) The column vectors of form an orthonormal
basis for .
( )
( )
( ) , , preserve inner product
( ) preserve norm
( ) preserve angle
n
i Q
ii Q Q I QQ
iii Q Q
iv Qx Qy x y
v Qx x
vi
![Page 84: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/84.jpg)
Theorem 5.5.6: If the columns of form an orthonormal set in , then and the least squares solution to is
This avoid computing matrix inverse .
m
m nA IAA
Ax b
1x A A A b A b
![Page 85: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/85.jpg)
Theorem 5.5.7 & 5.5.8:
Let be a subspace of an inner product
space and let . Let be
an orthonormal basis for .
If , where ,
then
1 2, , , nx x x
S
x V
(i)
(ii) - - in .
p x S
y x p x y p S
VS
1
n
i ii
p c x
, for each i ic x x i
The vector is said to be t projectionhe of onto .p x S
![Page 86: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/86.jpg)
Cor5.5.9:
Let be a subspace of and
If be an orthonormal basis for
and then
the projection of onto is .
pf:
1 ku u
S
S
1 ,kU u u
m
Sb
p
p UU b
1 1 2 2
1 1
2 2
From Thm.5.5.8, ... ,
where
Therefore, .
k k
T
TT
Tk k
p c u c u c u Uc
c u b
c u bc U b
c u b
p UU b
.mb
![Page 87: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/87.jpg)
Note: Let columns of be an
orthonormal set
1 kU u u
1
1
1
k
k
k
i ii
u
p UU b u u b
u
u b u
(i) The projection of onto ( ) is the
sum of the projection of onto each .
(ii) The matrix is called the projection
matrix onto .
i
T
b R U
b u
UU
S
![Page 88: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/88.jpg)
Example 7: Let
Find the vector in that is closet to
Sol: 5,3,4 .w
, ,0 , .S x y x y p
S
1 2
1 0
Clearly, , is a basis for . Let 0 1 ,
0 0
Thus,
1 0 0 5 5
0 1 0 3 3
0 0 0 4 0
e e S U
p UU w
1 1
2 21 1
: , what is ?2 2
0 0
THW U UU
![Page 89: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/89.jpg)
Approximation of functions
Example 8: Find the best least squares approximation to
on by a linear function .
Sol:
xe 1,0
2
2( )
12 2
0
( . ., Find g( ) 0,1 ( ) min ( ) ,
where , .)
x x
p x Pi e x P e g x e p x
f f f f dx
2
1
0
1 2
(i) Clearly, 1, 0,1 ,but 1, is not orthonormal.
(ii) seek a function of the form , ( ) 1
1 1 1, ( ) 0
2 21 1
2 12
1 1, 12( ) for
2
span x P x
x a x a
x a x a dx a a
x
u u x
2m an orthonormal set of 0,1 .P
![Page 90: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/90.jpg)
Sol:
1
1 1 0
1
2 2 20
1 1 2 2
(iii) c , 1
c , 3 3
Thus, the projection
( )
1 ( 1) 1 3(3 )( 12( ))
2
x x
x x
u e e e
u e u e dx e
p x c u c u
e e x
(4 10) 6(3 )
is the best linear least square approximation to on 0,1 .x
e e x
e
![Page 91: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/91.jpg)
Approximation of trigonometric polynomialsFACT: forms an orthonormal set in with respect to the inner product
Problem: Given a continuous 2π-periodic function , find a trigonometric polynomial of degree n
which is a best least squares approximation to .
1,cos ,sin
2nx nx n N
,0 C
dxxgxfgf )()(
1,
)(xf
0
1
( ) cos sin2
n
n k kk
at x a kx b kx
)(xf
![Page 92: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/92.jpg)
Sol: It suffices to find the projection of onto
the subspace
The best approximation of
has coefficients
)(xf
1,cos ,sin 1, ,
2span kx kx k n
( )nt x
0
1 1, ( )
2 2
1,cos ( )cos
1,sin ( )sin
k
k
a f f x dx
a f kx f x kxdx
b f kx f x kxdx
![Page 93: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/93.jpg)
Example: Consider with inner product of
(i) Check that is orthonormal
(ii) Let
,0 C
dxxgxfgf )()(
2
1,
0, 1, ,ikxe k n
1( )
21
( )2
nikx
n kk n
ikxk
k k
t c e
c f x e dx
a ib
Similarly, k kc c
![Page 94: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/94.jpg)
(iii)
(iv)
cos sin
ikx ikxk k
k k
c e c e
a kx b kx
0
1
cos sin2
nikx
n kk n
n
k kk
t c e
aa kx b kx
![Page 95: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/95.jpg)
5-6 Gram-Schmidt Orthogonalization Process
![Page 96: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/96.jpg)
Cram-Schmidt Orthogonalization ProcessQuestion: Given an ordinary basis ,
how to transform them into an orthonormal
basis ?
1 2, ,..., nx x x
1 2, ,..., nu u u
![Page 97: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/97.jpg)
Given ,Clearly
Clearly,
Similarly,
Clearly, We have the next result
1 nx x
1 11
1u x
x 1 1{ } { }span u span x
1 2 1 1 2 2 12 1
1, , ( )p x u u u x p
x p
1 2 1 2 1 2& { , } { , }u u span x x span u u
2 3 1 1 3 2 2
3 3 23 2
, ,
1( )
p x u u x u u
and u x px p
3 1 3 2 1 2 3 1 2 3, & { , , } { , , }u u u u span x x x span u u u
1u
1p
2x
![Page 98: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/98.jpg)
Theorem5.6.1: (The Gram-Schmidt process)
H. (i) Let be a basis for an inner
product space .
(ii)
C. is an orthonormal basis. 1 nu u
1 nx x
V
1 11
1 11
11
1,
1, 1, , 1
,
K K KK K
K
K K j jj
u xx
u x p K nx p
where p x u u
![Page 99: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/99.jpg)
Example: Find an orthonormal basis for with
inner product given by
, where
Sol: Starting with a basis
3P
),()(,3
1i
ii xgxPgP
.1&0,1 321 xxx
2,,1 xx
1 2 1
1 2
11 1 1 11 1
Let , ,..., be the projection vectors defines in Thm. 5.6.1, and
let , ,..., be the orthonormal basis of ( ) derived from the
Gram-Schmidt process.
Define
n
n
kk
p p p
q q q R A
r a a r q
r
1 for 2,...,
and for 1,..., 1 by the Gram-Schmidt process.
k k
Tik i k
a p k n
r q a i k
![Page 100: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/100.jpg)
Theorem5.6.2: (QR Factorization)
If A is an m×n matrix of rank n, then A
can be factored into a product QR, where Q
is an m×n matrix with orthonormal columns
and R is an n×n matrix that is upper triangular
and invertible.
![Page 101: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/101.jpg)
Proof. of QR-Factorization
1 2 1
1 2 1
11 1
Let , ,..., be the projection vectors defined in Thm.5.6.1,
and let , ,..., be the orthonormal basis of ( ) derived from
the Gram-Schmidt process.
Define
n
n
kk k k
p p p
q q q R A
r a
r a p
1
1 11 1
2 12 1 22 2
1 1
and for 1,... -1 for 2,...,
By the Gram-Schmidt process,
Tik i k
n n
r q a i kk n
a r q
a r q r q
a r q
... nn nr q
![Page 102: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/102.jpg)
Proof. of QR-Factorization (cont.)
1 2
11 12 1
22 2
If we set ( , ,..., ) and define to be the upper triangular matrix
0 ,
0 0
then the th column of the product wi
n
n
n
nn
Q q q q R
r r r
r rR
r
j QR
1 1 2 2
1 2
ll be
... for 1,... .
Therefore,
( , ,..., )
j j j jj j j
n
Qr r q r q r q a j n
QR a a a A
![Page 103: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/103.jpg)
Theorem5.6.3:
If A is an m×n matrix of rank n, then the
solution to the least squares problem
is given by , where Q and R are the
matrices obtained from Thm.5.6.2. The solution
may be obtained by using back substitution to solve .
Ax b
1x̂ R Q b
x̂
ˆRx Q b
![Page 104: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/104.jpg)
Proof. of Thm.5.6.3
ˆLet be the solution to the leaset squares problem
ˆ
ˆ
ˆ ( ) ( ) ( )
ˆ ( )
TAT T
T T
T T
I
x
Ax b
A Ax A b
QR QRx QR b QR Factorization
R Q Q R
1
ˆ ( is invertible)
ˆ ˆ or
T T
T T T
T
x R Q b
R Rx R Q b R
Rx Q b x R Q b
![Page 105: Chapter 5 Orthogonality. Outline Scalar Product in R n Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649d375503460f94a109d5/html5/thumbnails/105.jpg)
Example 3: Solve
By direct calculation,
1
2
3
1 2 1 1
2 0 1 1
2 4 2 1
4 0 0 2bA
x
x
x
R
Q
QRA
200
140
125
1
2
2
4
2
4
1
2
4
2
2
1
5
1
1
1
2
Q b
The solution can be obtained from
5 2 1 1
0 4 1 1
0 0 2 2