CHAPTER FIVECHAPTER FIVEOrthogonalityOrthogonality
Why orthogonal?Least square problemAccuracy of Numerical computation
Least square problemLeast square problem
Motivation: Curve fittingProblem formulation:
Given ,
Find such that
nmA mb
0x
xAbxAbnx
o
min
OutlinesOutlines
Results ofOrthogonal subspaceInner product spaces Least square problemsOrthonormal setsGram-Schmidt Orthogonalization process
n
The scalar product inThe scalar product in
Def: Let , .the
inner product (or scalar product)
of and is defined to be
The norm of is defined as
n
ny x
x y
n
iii yxyxyxyx
1
,
xxx ,
x
Theorem5.1.1: Let . Then
pf: By the law of cosines,
Note: If is the angle between , then
Thus
Def:
nyx &
cosyxyx
cos2222
yxyxxy
222
2
1cos xyyxyx
222
2
1iiii xyyx
ii yx yx
yx&
y
y
x
x
cos 11
yx
yx
0 yxyx
Cor: (Cauchy-Schwartz Inequality)
Let . Then
Moreover , equality holds
for some
nyx &
yx
.yxyx
Scalar and Vector ProjectionsScalar and Vector ProjectionsLet . Then the quantify is the scalar projection of onto and the vector is called the vector projection of ontoIf ,then the scalar projection of onto is and the vector projection of onto is
cosuxux
1u
x
x
x
u
u
uuxu
x
x y
y
1y
y
yx
yyy
yx
y
y
y
yx
Example: Find the point
on the line that
is closest to the point
(1,4)
Sol: Note that the vector is on the line
Then the desired point is
xy3
1
1
3w xy
3
1
7.0
1.2w
ww
wv
v
Q
xy3
1
4.1
Example: Find the equation of the plane
passing through and
normal to
Sol:
4,3,2
3,1,2
0
3
1
2
4
3
2
z
y
x
0341322 zyx
Example: Find the distance form
to the plane
Sol: a normal vector to
the plane is
desired distance
0,0,2P
022 zyx
2
2
1
3
2
v
vP
v
P
Orthogonal SubspaceOrthogonal Subspace
Def: Let be two subspace
of . We say that if
, and .Example: Let
but does not orthogonal to
YX &
n YX
0 yx Xx Yy
},{ 1espanX },{ 2espanY },{ 32 eespanZ
.& ZXYX
Y Z
Def: Let be a subspace of .
Then the orthogonal complement
of is defined as
Example: In ,
Y n
Y
},0{ FyyxxY n
3 },{}{ 321 eespanYespanY
}{},{ 321 espanXeespanX
Lemma: (i) Two subspaces
(ii) If is a subspace of ,then is
also a subspace of .
Pf: (i) If
(ii) Let
and
}0{ YXYX
Xn
Xn
YXx
002
xxxx
Xyy21
&
Xx
xyxyxyy 2121
0
Four Fundamental SubspacesFour Fundamental SubspacesLet
for some
for some
It will be shown later that
nmA
mnAor :xAxmnA
AN nn xAx 0
AN
AR
mm xAx 0
xAbb m { mnx }
AR xAbb n { nmx }
ARAN ARAN
ARANn
ARANm
Theorem5.1.2: Let . Then and pf: Let and _____(1) for some _____(2)
________(3) Also, if _____(4)
Similarly,
nmA ARAN ARAN
ANx ARy
0:, xiA :,&
1
iAym
ii
0:,)1(
iAxyx i
ANARzA 0
ARAN)4)(3(
mi ,1
i
ARAN
ARAN
miiAZARz ,1,0:,
ARARAN
Example: Let
Clearly,
00
21
02
01AA
1
0spanAN
1
2spanAN
2
1spanAR
0
1spanAR
ARAN
ARAN
Theorem5.2.2: Let be a subspace of ,
then (i)
(ii) If is a basis for and
is a basis for ,then
is a basis for .
pf: If The result follows
Suppose . Let
and
nnSS dimdim
}{ 1 rxx }{ 1 nr xx
n
S
SS
nSS }0{}0{S
)()(
XrankrXrank
rnrxxX )( 1
SXR )(
)()(1.2.5 XNXRS
Theorem
rnXNSTheorem
4.6.3
)(dim)dim(
}{ 1 rxx
To show that is a basis for ,
It remains to show their independency.
Let . Then
Similarly,
This completes the proof
n
01
n
iii xc Sx
011
r
iii
n
iii xcxxcx
ricxc i
r
iii 1,00
1
Sy
01
n
riii
n
ii xcyxcy
nricxc i
n
riii ,1,00
1
}{ 1 nxx
Def: Let are two subspaces
of . We say that is a
direct sum of ,denoted by
, if each
can be written uniquely as
a sum , where Example: Let
Then
but
VU &
W W
VU &
VUW
Ww
vu VvUu &
1
0
0
,
1
0
0
0
1
0
,
0
1
0
0
0
1
spanWspanVspanU
,3 WU VU 3 VU 3
Theorem5.2.2: If is a subspace of ,
then
pf: By Theorem5.2.2,
To show uniqueness,
Suppose
where
nS SSn
SSn
.&,, SvSuvuxx n
2211 vuvux SvvSuu 2121 ,&,
SSvvuu 1221
}0{SS
1221 & vvuu
Theorem5.2.4: If is a subspace of ,
then
pf: Let
If
(Why?)
S nSS )(
rS )dim(
rSTheorem
)dim(2.2.5
0 yxSx Sy
Sx
SS
SS
Remark: Let . i.e. , Since
and
are bijections .
nmA mnA :
ARANn
ArankArank )( ArankAnullityn ArankAnullity
ARARA :
ARARA :
Let nmA nA :
mA :
m
n
A
A
A
A
AN
AN
AN
AN
bijection
bijection
Cor5.2.5:
Let and . Then
either
(i)
or
(ii)
pf: or
Note that
nmA mb
bxAx n
my0&0 byyA
)(ARb )(ARb
0)()()( byANyANARb
0&0 byyAy m
)()( Tn ANAR
)(AR
b
)( AN
Example: Let . Find
The basic idea is that the row space and the sol. of are invariant under row operation.Sol: (i) (Why?)
(ii) (Why?)
(iii) Similarly,
and
(iv) Clearly,
431
110
211
A )(),(),(),( ARANARAN
bxA
000
110
101
~ r
row
AA
1
1
0
1
0
1
)( spanAR
0&00 3231 xxxxxAr
1
1
1
)( spanAN
000
210
101
~row
A
2
1
0
1
0
1
)( spanAR
1
2
1
)( spanAN
)()(& ARANARAN
Example: Let
(i)
and
(ii) The mapping
is a bijection
and
(iv) What is the matrix representation for ?
23:0
0
3
0
0
2
A
0
1
0
0
0
1
1
0
03 spanspanARAN
2)( AR
)(:)(
ARARAAR
0
3
2
02
1
2
1
x
x
x
x
)(:1
)(
ARARAAR
03
12
1
2
1
2
1 y
y
y
y
)( ARA
Linear Product SpaceLinear Product Space
A tool to measure the
orthogonality of two vectors in
general vector space
Def: An inner product on a
vector space is a function
Satisfying the following conditions:
(i) with equality iff
(ii)
(iii)
V
)(:, orCFVV
0, xx 0x
xyyx ,,
zyzxzyx ,,,
Example: (i) Let
Then is an inner product of
(ii) Let , Then is an
inner product of
(iii) Let and then
is
an inner product of
(iv) Let , is a positive function and
are distinct real numbers. Then
is
an inner product of
.,10&, niwyx in
ii
n
ii yxwyx
1
, nij
n
jij
m
i
baBA
11
,nmBA ,nm
].,[)(,, 0 baCxwgf 0)( xw
b
a
dxxgxfxwgf )()()(,
].,[0 baC
ngp , )(xw
nxx 1
)()()(,1
ii
n
ii xgxPxwgp
n
Def: Let be an inner
product of a vector space
and .
we say
The length or norm of is
,V
Vvu ,
0; vuvuv
vvv ;
Theorem5.3.1: (The Pythagorean Law)
pf:
222vuvuvu
22vu
vuvuvu ,2
vvvuuu ,,2,
u
v
vu
Example: Consider with inner product
(i)
(ii)
(iii)
(iv) (Pythagorean Law)
or
]1,1[0 C
1
1)()(, dxxgxfgf
xxdxx 101,11
1
212111,11
1 dx
32
3
2,
1
1 xxdxxxx
3
8
3
2211
222 xx
3
8)1(1,11
21
1
2 dxxxxx
Example: Consider with inner product
It can be shown that
(i)
(ii)
(iii)
Thus are
orthonormal set.
]1,1[0 C
dxxgxfgf )()(
1,
0sin,cos mxnx
mnnxmx cos,cos
mnnxmx sin,sin
Nnnxnx sin,cos
Example: Let
and let
Then not orthogonal
to
nmBA ,
AAA
baBA
F
ij
m
i
n
jij
,
,1 1
4
0
1
3
3
1
,
3
2
1
3
1
1
BA
ABA 6,
B6,5
FFBA
Def: Let be two vectors in an
inner product space . Then
the scalar projection of onto is
defined as
The vector projection of onto is
0& vu
V
u v
v
vu
v
vu
,,
u v
vvv
vu
v
vP
,
,
Lemma: Let be the vector projection
of onto . Then
for some
pf:
Pv &0u v
vkuPui
PPui
)(
)(
trivialii
PuP
vv
vu
vv
vu
PPuPPuPi
)(
0,
,
,
,
,,,)(
22
u
v
P
Pu
k
Theorem5.3.2: (Cauchy-Schwarz Inequality)
Let be two vectors in an
inner product space . Then
Moreover, equality holds are linear dependent.
pf: If
If
Equality holds
i.e., equality holds iff are linear dependent.
vu&
V
vuvu ,
vu&
thenv
trivialv
,0
,0
222
2
,
,PuuP
vv
vu TheoremnPythagorea
22vu
22222
, Puvvuvu
vvv
vuPu
,
,orv ,0
vu&
uPu
vP
Note:
From Cauchy-Schwarz Inequality.
This, we can define as the
angle between the two vectors
vu
vu
vu
vu
,cos
,0!
1,
1
.& vu
Def: Let be a vector space
A fun is said
to be a norm if it satisfies
with equality
scalar
V
:
vv
FV
0 v
wvwviii
vvii
vi
)(
,)(
0)(
Theorem5.3.3: If is an inner product
space, then
is a norm on
pf: trivial
Def: The distance between is defined
as vu
V
V
vu&
vvv
Example: Let . Then
is a norm
is a norm
is a norm for any
In particular ,
is the
euclidean norm
nx
Pn
i
P
iP
ini
xxiii
xxii
1
1
1
)(
)( max
n
iixxi
11
)(
1P
xxxxn
ii ,
1
2
2
Example: Let . Then
3
5
4
x
5
25
12
2
1
x
x
x
Example: Let
Thus,
However,
(Why?)
2
4&
2
121 xx
0, 21 xx2
221
2
22
2
225205 xxxx
1620
1642
21
2
2
2
1
xx
xx
Example: Let
Then
12
xxB
1B2B B
1 1 1
Least square problemLeast square problem A typical example:
Given
Find the best line
to fit the data .
or
or find such that
is minimum
Geometrical meaning :
niy
x
i
i ,1,
xccy 10
nn y
y
y
c
c
x
x
x
solve2
1
1
02
1
1
1
1
bxA
10 ,ccbxA
xccy 10
),( nn yx
),( 11 yx
Least square problem:
Given
then the equation
may not have solutions
The objective of least square problem is
trying to find such that
has minimum value
i.e., find satisfying
,& mnm bA
bxA
)()(.,. ARAColbei
x
xAb
x
xAbxAbnx
min )(AR
b
xA
Preview of the results:
It will be shown that
Moreover,
If columns of are Linear independent .
byPb
ARP
ARy
)(min
)(!
ybPbARy
)(min
bAxAA
xAbA
PbA
ANARPb
0
0
)()(
bAAAx
1
A
b
)(ARP
Theorem5.4.1: H. Let be a subspace of
C. (i)
for all
(ii)
pf:
where
If
Since the expression is unique,
result (i) is then proved .
(ii) follows directly from (i) by noting that
S m
Pbyb
SPb m
!,
}{\ PSy
SPbbybP
Symin
zPb
SSi m
)(
SzSP &}{\ PSy
2
0
2
2
2
yPPbyPPbybTheoremnPythogorea
SSz
SzPb
zPb
P
b
S
Question: How to find which solves
Answer: Let
From previous Theorem , we know that
normal equation
x
?min xAbbxAnx
nmA
0
0)(
)()(
xAAbA
PbA
ANARPb
)(AR
xAP
b
Theorem5.4.2: Let and
Then the normal equation . Has
a unique sol .
and is the unique least square
sol . to
pf: Clearly, is nonsingular (Why?)
is the unique sol. To normal equation .
is the unique sol . To the
least square problem (Why?)
( has linear independent columns)
nmA .)( nArank bAxAA
bAAAx
1
x
bxA AA
x
x
A
Note: The projection vector
is the element of that
is closet to in the least square
sense . Thus, The matrix
is called the
Projection matrix (that project any
vector of to )
bAAAAxAP
1
)(AR
b
AAAAP1
)(ARm
b
)(AR
P
Example: Suppose a spring obeys the Hook’s law
and a series of data are taken (with measurement
error) as
How to determine ?
sol: Note that
is inconsistent
The normal equation. is
The least square sol.
KxF
11
8
7
5
4
3
x
F
8
5
3
11
7
4
811
57
34
Kor
K
K
K
8
5
3
1174
11
7
4
1174 K
K
726.0K
Example: Given the data
Find the best least square fit by a linear function.sol: Let the desired linear function be The problem be comes to find the least square sol. of
Least square sol.
The best linear least square fit is
5
6
4
3
1
0
y
x
xccy 10
x
Ab
c
c
1
0
6
3
0
1
1
1
5
4
1
3
23
4
1
0 bAAAc
c
xy3
2
3
4
Example: Find the best quadratic least square
fit to the data
sol: Let the desired quadratic function . be
The problem becomes to find the least square
sol . of
least square sol .
the best quadratic least square fit is
25.0
25.0
75.2
2
1
0
c
c
c
4
3
4
2
2
1
3
0
y
x
2210 xcxccy
2
1
0
9
4
1
0
3
2
1
0
1
1
1
1
4
4
2
3
c
c
c
225.025.075.2 xxy
Orthonormal SetOrthonormal Set
Simplify the least square sol.
(avoid computing inverse)Numerical computational stability
Def: is said to be an orthogonal set in
an inner product space if
Moreover, if
, then
is said to be orthonormal
nvv 1
V
jiallforvv ji 0,
ijji vv ,
nvv 1
Example:
is an
orthogonal set but not
orthonormal
However ,
is orthonormal
1
5
4
,
3
1
2
,
1
1
1
321 vvv
42,
14,
33
32
21
1
vu
vu
vu
Theorem5.5.1: Let be an orthogonal
set of nonzero vectors in an inner
product space . Then they are
linear independent
pf: Suppose
is linear independent .
nvv 1
V
01
i
n
iiVc
nj
jjj
ijiiij
vv
njc
VVc
VVcVcV
1
,1,0
,
,0,
Example:
is an
orthonormal set of
with inner product
Note: Now you know the meaning what one
says that
Nnnsnx sincos,
2
1
,0 C
dxxgxfgf )()(
1,
xx sincos
Theorem5.5.2: Let be an
orthonormal basis for an inner
product space . If
then
pf:
nuu 1
V i
n
iiucv
1
vuc ii ,
jij
n
jj
ji
n
jjj
n
jjii
cc
uucucuvu
1
11
,,,
Cor: Let be an orthonormal
basis for an inner product space .
If and ,
then
pf:
V
nuu 1
n
iiiuau
1
n
iiiubv
1
n
iiibavu
1
,
n
iii
Theorem
n
iii
n
iii
ba
vua
vuavu
1
2.5.5
1
1
,
,,
Cor: (Parseval’s Formula)
If is an orthonormal
basis for an innerproduct space
and , then
pf: direct from previous corollary
nuu 1
V
n
iiiucv
1
n
iicv
1
2
Example:
and from
an orthonormal basis for .
If , then
and
2
12
1
1u
2
12
1
2u
22
2
1
x
xx
2, 21
2
xxux
,
2, 21
1
xxux
221
121
2.5.5
22u
xxu
xxx
Theorem
22
21
2
21
2
212
22xx
xxxxx
Example: Determine without
computing antiderivatives .
sol:
and is an orthonormal set of
xdx
4sin
xx
x
xxxxdx
2cos2
1
2
1
2
1
2
2cos1sin
sinsin,sinsin
2
22224
x2cos,2
1 ,0 C
4
3
2
1
2
1
sinsin
22
224
xxdx
Def: is said to be an orthogonal matrix if the column vectors of form an orthonormal set in Example: The rotational matrix
and the elementary reflection
matrix are orthogonal
matrix .
nmQ
nQ
cossin
sincos
cossin
sincos
Properties of orthogonal matrix:
at be orthogonal . Then
The column vectors of form an
orthonormal basis for
Preserve inner product
preserve norm
preserve angle .
nmQ
)(
)(
,,)(
)(
)(
)(
22
1
vi
xxQv
yxyQxQiv
QQiii
IQQii
Qin
Note: Let the columns of form
an orthonormal set of .Then
and the least
square sol to is
This avoid computing matrix inverse .
nA
IAA
bxA
bAbAAAx
1
Cor5.5.9:
Let be a nonzero subspace of
and is an orthonormal
basis for . If
then the projection of onto is
pf:
Kuu 1
S
S ,1 KuuU
m
SbP
bUUP
bUU
bUUUUP
Note: Let columns of be
an orthonormal set
The projection of onto
is the sum of the projection
of onto each .
KuuU 1
b
u
u
uubUU
K
K
1
1
i
K
ii ubu
1
iu
b
b
)(UR
Example: Let Find the vector in that is closet to
Sol: Clearly is a basis for . Let Thus
Hw: Try
What is ?
.4,3,5 w
yxyxS ,0,,P S
21,ee
S
0
1
0
0
0
1
U
0
3
5
4
3
5
000
010
001
wUUP
UU
02
12
1
02
12
1
U
Approximation of functionsApproximation of functions
Example: Find the best least square
approximation to on by a linear function .
Sol: (i) Clearly ,
but is not orthonormal
(ii) seek a function of the form
By calculation
is an orthonormal set of
(iii)
Thus the projection .
is the best linear least square approximation to on
xe 1,0
1
0
22
2)(20
20
,
)(min)(
1,0)(.,.
2
dxffffwhere
xPexPe
PxPFindeix
PxP
x
1,0,1 2Pxspan x,1
1)( axax
1,0xe
xeexee
ucucxP
)3(6)104())2
1(12)(3(31)1(
)( 2211
edxeueu
eeeu
xx
xx
33,
1,
1
0 22
1
01
2
10
2
1)(,1
1
0 aadxaxax
12
1
2
1x
1,02P
)
2
1(12,1 21 xuu
Approximation of trigonometric polynomialsApproximation of trigonometric polynomials
FACT: forms an
orthonormal set in with respect
to the inner product
Problem: Given a periodic function ,
find a trigonometric polynomial of degree n
which is a best least square approximation to .
Nnnxconx sin,,
2
1
,0 C
dxxgxfgf )()(
1,
2 )(xf
n
KKKn KxbKxa
axt
1
0 sincos2
)(
)(xf
Sol: It suffices to find the projection
of onto the subspace
The best approximation of
has coefficients
)(xf
nKKxconKxspan ,,1sin,,
2
1
tn
dxKxxfKxfb
dxKxxfKxfa
dxxffa
K
K
sin)(1
sin,
cos)(1
cos,
)(2
1
2
1,0
Example: Consider with
inner product of
(i) Check are orthonormal
(ii) Let
Similarly
(iii)
(iv)
,0 C
dxxgxfgf )()(
2
1,
nKeiKx ,,1,0
)(2
1
)(2
1
KK
iKxK
n
nK
iKxKn
iba
dxexfC
eCt
KK CC
KxbKxa
eCeC
KK
iKxK
iKxK
sincos
n
KKK
n
nK
iKxKn
KxbKxaa
eCt
1
0 sincos2
Cram-Schmidt Orthogonalization Cram-Schmidt Orthogonalization ProcessProcess
Question: Given a set of linear independent
vectors,
how to transform them into
orthogonal ones while
preserve spanning set.?
Given
,Clearly
Clearly
Similarly,
Clearly
We have the next result
Kxx 1
1
11 x
xu
}{}{ 11 xspanuspan
12
1221121 ,,
Px
PxuuuxP
},{},{& 212121 uuspanxxspanuu
23
233
2231132 ,,
Px
Pxuand
uuxuuxP
},,{},,{&, 3213212313 uuuspanxxxspanuuuu
1u 1P
2x
Theorem5.6.1: (The Cram-Schmidt process)
H. (i) be a basis for
an inner product space
(ii)
C. is an orthonormal basis nuu 1
nxx 1
V
j
K
jjKK
KK
KKK
uuxPwhere
nKPx
Pxu
xxu
11
1
11
1
11
,
1,,1,
,
Example: Find an orthonormal basis for
with inner product given by
where
Sol: Starting with a basis
3P
),()(,3
1i
ii xgxPgP
.1&0,1 321 xxx 2,,1 xx
32
32
03
2
22,
3
1
3
1,
2
03
1
3
1,
3
1
1
1
2
22
22
3
222
1
12
1
1
x
Px
Pxu
xxxxP
xPx
Pxu
xP
u
QR-DecompositionQR-DecompositionGiven
Let _______(1)
_________________(2)
_______________________(3)
Define
Where has orthonormal columns and is upper-triangular
naa 1
1111111 graar
1121121 , grggaP
1222 Par
2221122221
)3)(2(
222
122 grgrgrPa
r
Pag
i
K
iKii
K
iKiK grgagP
1
11,
1
11 ,
1
KKKK Par
KKK
K
iKiK
KK
KKK grrQ
r
Pag
1
11,
1
nnijnmn rRggQ
,1
QRAQ R
To solve with
Then, the example can be solved
By backsubstitution without finding
Inverse (if is square)
bxA nArankA nm )(&
bQxR
bxQR
A
Example: Solve
By direct calculation,
The solution can be obtained from
bA
x
x
x
2
1
1
1
0
2
1
1
0
4
0
2
4
2
2
1
3
2
1
R
Q
QRA
200
140
125
1
2
2
4
2
4
1
2
4
2
2
1
5
1
2
1
1
bQ
2
1
1
200
140
125
Top Related