CHAPTER SIX Eigenvalues. Outlines System of linear ODE (Omit) Diagonalization Hermitian matrices...
-
Upload
oswin-edwards -
Category
Documents
-
view
219 -
download
3
Transcript of CHAPTER SIX Eigenvalues. Outlines System of linear ODE (Omit) Diagonalization Hermitian matrices...
CHAPTER SIXCHAPTER SIX
EigenvaluesEigenvalues
Outlines
System of linear ODE (Omit)DiagonalizationHermitian matricesOuadratic formPositive definite matrices
MotivationsTo simplify a linear dynamics such
that it is as simple as possible.To realize a linear system characteristics
e.g.,
the behavior of system dynamics.
xAx
zecharacteripofroot
bapeqch
byyay
)(
0)(..
0
2
Example: In a town, each year 30% married women get divorced 20% single women get married In the 1st year, 8000 married women 2000 single
women. Total population remains constant
be the women numbers at year i,
where represent married & single women
respectively.
si
mi
iW
WWLet
si
mi WW &
ii WWW
8.03.0
2.07.0&
2000
800010
If
Question: Why does converges?
Why does it converges to the same limit even
when the initial condition is different?
12,&6000
4000,,
4000
6000
2000
8000,
8.03.0
2.07.0
12121
01
nWWWW
WWAWW
n
iii
14,
6000
4000
0
10000
14
14
0
iWW
W
W
i
iW
Ans: Choose a basis
Given an initial for some
for example
Question: How does one know choosing such a basis?
1
1&
3
221 xx
2211 2
1& xxAxxA
1
1)4000(
3
22000
2000
8000
221100 xCxCWW 21 &CC
nasxCxACxACWA nnn1122110
Def: Let . A scalar is said to be an
eigenvalue or characteristic value of A if
such that
The vector is said to be an eigenvector
or characteristic vector belonging to .
is called an eigen pair of A.
Question: Given A, How to compute eigenvalues
& eigenvectors?
xxA
nnFA
0x
x
),( x
is an eigen pair of
is singular
Note that, is a polynomial, called
characteristic polynomial of A, of degree n in
Thus, by FTA, A has exactly n eigenvalues including
multiplicities. is a eigenvector associated with
eigenvalue while is eigenspace of A.
0)det(
),(
0,0)(
0,
IA
IA
xxIA
xxxA
nnFA
.
)det()( AIp
0\)( AINx
),( x
)( IAN
Example: Let
are eigenvalues of A. To find eigenspace of 2:( i.e., )
3&2
)3)(2(65
11
24det
)det()(
11
24
2
AIp
A
1
1)2(
0
0
11
220)2(
2
1
spanIAN
x
xxIA
)2( IAN
To find eigenspaces of 3(i.e., )
Let
1
2)3(
0
0
21
210)3(
2
1
spanIAN
x
xxIA
)3( IAN
30
02
11
21
1App
p
Let . Then
is an eigenvalue of A.
has a nontrivial solution.
is singular.
loses rank.
nIANullityvii
IAvi
IAv
IAiv
IANiii
xIAii
i
)()(
)(
0)det()(
)(
0)()(
0))((
)(
nnFA
Let .
If is an eigenvalue of A with eigenvector
Then
This means that is also an eigen-pair of
A.
xxxAxA
xxA
nnA
.xC
),( x
Let .
Where are eigenvalues of A.
(i) Let
(ii) Compare with the coefficient of , we
have
nnA
)()det()(1
i
n
iAIp
n 1
n
iiA
1
det0
1n)(
11
Atracean
iii
n
ii
Theorem 6.1.1: Let
Then and consequently A & B have the
same eigenvalues.
Pf: Let for some nonsingular matrix S.
)()det(
)det()det()det(
)(det
)det(
)det()(
1
1
1
A
B
PAI
SAIS
SAIS
ASSI
BIP
BA ~
)()( BA PP
ASSB 1
Diagonalization
Goal: Given find nonsingular matrix S
a diagonal matrix.
Question1: Are all matrices diagonalizable?
Question2: What kinds of A are diagonalizable?
Question3: How to find S if A is diagonalizable?
nnFA
DASS 1
NOT all matrices are diagonalizable
e.g., Let
If A is diagonalizable
nonsingular matrix S
00
10A
2
11
0
0
d
dASS
00
0
0
0)det(
0)(
1
2
1
21
21
21
Sd
dSA
dd
ddA
ddAtrace
To answer Q2,
Suppose A is diagonalizable.
nonsingular matrix S
Let
are eigenpair of A for
This gives a condition for and diagonalizability and a way
to find S.
nSSS 1
ni ,,1
SDAS
)( 11
ndddiagDASS
nSASA 1 nn SdSd 11
ii Sd
Theorem 6.3.2: Let is diagonalizable
A has n linear independent eigenvectors.
Note : Similarity transformation
Change of coordinate
diagonalization
)()( EE
WL
A
B
1 SASB
)()( EE
WL
SI
nnFA
xPy
SI
Theorem6.3.1: If are distinct eigenvalues of a
matrix A with corresponding eigenvectors
, then are linear independent.
Pf: Suppose are linear independent
not all zero
Suppose
are distinct.
are linear independent.
.... golw
0)(2
iii
K
ixCIA
n 1
0)(2
11
K
ii xC
nn
01
i
K
ii xC
Kxx 1 Kxx 1
Kxx 1
KCC 1
01 C
K
i
xx
C
x
1
1
1
*0
&0
Remarks: Let , and
(i) is an eigenpair of A for
(ii) The diagonalizing matrix S is not unique because
Its columns can be reordered or multiplied by
an nonzero scalar
(iii) If A has n distinct eigenvalues , A is diagonalizable.
If the eigenvalues are not distinct , then may or may not
diagonalizable depending on whether or not A has n
linear independent eigenvectors.
(iv)
1SDSA )( 1 ndiagD nSSS 1
ii S ni ,,1
11
1 SdiagSSSDA nkk
Example: Let
For
For
Let
112
202
213
A
100
010
0001SDS
1,1,00det AI
1
1
1
)0(,0 SpanIAN
1
2
0
,
0
2
1
)(,1 SpanIAN
101
221
011
S
Def: If an matrix A has fewer than nlinear independent eigenvectors,we say that A is defective
e.g.
(i) is defective
(ii) is defective
10
11A
nn
00
10A
Example: Let A & B both have the same eigenvalues Nullity (A-2I)=1 The eigenspace associated with has only one dimension. A is NOT diagonalizable However, Nullity (B-2I)=2 B is diagonalizable
263
041
002
&
201
040
002
BA
4,2,2
2
001
020
000
)2(
rankIArank
1
063
021
000
)2(
rankIBrank
2
Question:
Is the following matrix diagonalizable ?
00
00
01
)(
00
10
01
)( BiiAi
The Exponential of a Matrix Motiration :
Motiration:The general solution of
is
The unique solution of
is
Question:What is and
how to compute ?
xAx cetx At)(
00 )( xtx
xAx
0)( 0)( xetx ttA
AteAte
Note that
Define
0 !i
ia
i
ae
......!2!
2
0
AAI
i
Ae
i
iA
Suppose A is diagonalizable with
1
00
)(!
SDS
i
Ae
i
i
i
iAt
1SDSA
kSSDA kk ,1
1 SSeDt
1
0
01
S
e
e
St
t
n
Example: Compute
Sol: The eigenvalues A are
with eigenvectors
0,1
1
3
1
221 xandx
00
01
11
32;1 DXXDXA
11
10
0
X
eXXXee DA
231
6623
ee
ee
Hermitian matrices : Let , then A can
be written as
where
e.g. ,
ii
iiA
274
32
21
11
74
32i
nnCA
iCBA
nnRCB ,
Let , then
e.g. ,
ii
ii
ii
iiH
273
42
274
32
nnCA TH AA
HHH
HHH
HH
ACAC
BABA
AA
Def:
A matrix A is said to be Hermitian if
A is said to be skew-Hermitian if
A is said to be unitary if
( → its column vectors form an orthonormal set in )
HAA
AAH
IUU H nC
Theorem6.4.1: Let Then
(i)
(ii) eigenvectors belonging to distinct eigenvalues are orthogonal
Pf : (i)Let be an eigenpair of A
(ii) Let and be two eigenpairs of A with
HAA RA
x,) õjjxjj2= xHAx = xHAHx = (xHAx)H = õöjjxjj2
) õ = õö ) õ 2 R(õ1;x1) (õ2;x2)
õ16=õ2) (õ1à õ2)xH1x2= õ1xH1x2à õ2xH1x2
= (õ1x1)Hx2à xH1 (õ2x2)
= xH1AHx2à xH1Ax2= 0
) xH1x2= 0) x1 ? x2
(i)
Theorem: Let Then
Pf : (i) Let be an eigenpair of A
is pure-imaginary
HAA
axisjA
x,
xxxAx
xAxxxxxHH
HHHH
AAH
Theorem6.4.3: (Schur`s Theorem) Let
Then unitary matrix U is
upper triangular
Pf : Let be an eigenpair of A with
Choose to be such that is unitary
Choose
Chose to be unitary
Continue this process , we have the theorem
nnCA AUU H
11,w 11 w
nww 2 nwwU 11
2
11
1
1
11
*0
*
nTn
T
H wAwA
w
w
AUU
222
211
2112
2
2
*0
*
0
01
ww
w
UAUUU
w
UH
HH
2w
)1()1(4
32
222
0
nn
Hww
0
0
0
2
1
2112
UUAUU HH
Theorem6.4.4: (Spectral Theorem)
If , then unitary matrix U
that diagonalizes A .
Pf : By previous Theorem , unitary matrix
, where
T is upper triangular .
T is a diagonal matrix
AAH
TAUUU H
TAUU
UAUAUUTH
HHHHH
AAH
Cor: Let A be real symmetric matrix .Then (i) (ii) an orthogonal matrix U is a diagonal matrixRemark : If A is Hermitian , then , by Th6.4.4 , Complete orthonormal eigenbasis
RA
AUU T
Example:
Find an orthogonal matrix U that diagonalizes A
Sol : (i)
(ii)
(iii)By Gram-Schmidt Process
The columns of
form an orthogonormal eigenbasis (WHY?)
021
232
120
ALet
TT
T
SpanAIN
SpanAIN
0,1,2,1,0,1)(
6
1,
6
2,
6
1)5(
)5()1()det()( 2 AIp5,1,1
TT
SpanAIN3
1,
3
1,
3
1,
2
1,0,
2
1)(
3
1
2
1
6
13
10
6
23
1
2
1
6
1
U
1,1,5 diagAUU T
Question:In addition to Hermitian matrices ,
Is there any other matrices possessing orthonormal
eigenbasis?
Note : If A has orthonormal eigenbasis
where U is Hermitian &
D is diagonal
HUDUA
HHHH
HHHH
HHHH
AAUUDUDU
UUDDDUUD
UDUUUDAA
Def: A is said to be normal if
Remark : Hermitian , Skew- Hermitian and Unitary matrices are all normal
HH AAAA
Theorem6.4.6: A is normal A possesses orthonormal eigenbasisPf : have proved By Th.6.4.3 , unitary U is upper triangular T is also normal Compare the diagonal elements of
T has orthonormal eigenbasis(WHY?)
""""
AUUT HHH AAAA
HH TTTT
HH TTTT &
2
1
2
2
2
2
2
1
2
2
1
2
1
2
11
nn
n
iin
n
jj
ii
n
jj
tt
tt
tt
jiTij ,0
Singular Value Decomposition(SVD) :
Theorem : Let with rank(A)=r
Then unitary matrices
With
Where
nmCA nmCU
rrr
nn RdiagCV 1&021 r
Hi
r
iii
H
nm
rrH
uu
VVUUVUA
1
212100
0
nrr
mrr
vvVvvV
uuUuuU
1211
1211
,
,
Remark:In the SVD
The scalars are called
singular values of A
Columns of U are called
left singular vectors of A
Columns of V are called
right singular vectors of A
HVUA
r 1
Pf : Note that , is Hermitian
& Positive semidefinite with
unitary matrix V
where
Define
(1)
(2)
Define (3)
Define is unitary
nnH CAA
rAArank H )(
H
rH VdiagVAA 00,22
1 021 r
)(211 )()(& rnnrnr VVVdiag
HH VVAA 1
2
1
00 22 AVAVAH
rmCAVU 1
11
)3)(1(r
H IUU 11
00
0,HVUA
00
02121 UUVVA
mmCUUU 212 )3)(2(
Remark:In the SVD The singular values of A are unique while U&V are not unique Columns of U are orthonormal eigenbasis for Columns of V are orthonormal eigenbasis for
is an orthonormal basis for
is an orthonormal basis for
is an orthonormal basis for
is an orthonormal basis for
HVUA r 1
AAHHAA
VUA
UUAV
H
00
0
nr vv 1
mr uu 1
ruu 1
rvv 1
)( HAN
)( HAR
)(AN
)(AR
rank(A) = number of nonzero singular values
but rank(A) ≠ number of nonzero eigenvalues
for example
0,0)(1)(
00
10
AbutArankA
Example : Find SVD of
Sol :
An orthonormal eigenbasis associate with
can be found as
Find U is orthogonal
A set of candidate for are
Thus
00
11
11
A
0,40,4)( 21 AAT
AAH
11
11
2
1V
TvAu )0,2
1,
2
1(
11
11
)()(, 32 VUAANuu TT
32 &uu
3311
2 ,)0,2
1,
2
1(
1euvAu T
00
00
02
withVUA T
Lemma6.5.2 : Let
be orthogonal . Then
Pf :
mmnm RQRA &
FFAQA
2
1
2
2
1
2
2
21
2
F
n
ii
n
ii
FnF
A
A
QA
QAQAQA
Cor : Let be
the SVD of A . Then
nnH RVUA
n
iiFF
A1
2
We`ll state the next result without proof :
Theorem6.5.3 :
H.(1) be the SVD of A
(2)
C :
nnH RVUA
kSrankRS nm )(
FS
n
kji
F
k
i
Tiii ASvuA
min1
2
1
Application : Digital Image Processing
p. 377
(especially efficient for matrix which
has low rank)
Quadratic Forms :
To classify the type of quadratic surface (line) Optimization : An application to the Calculus
Def : A quadratic equation in two variables x & y
is an equation of the form
02 22 feydxcybxyax
0),(),(
f
y
xed
y
x
dc
bayx
Standard forms of conic sections
(i)
(ii)
(iii)
(iv)
Note : Is there any difference between the eigenvalues
A of the quadratic form ?XAX T
circlery
xyxryx
2222
10
01)(
ellipsey
x
b
ayxb
y
a
x
11
0
01
)(1
2
2
2
2
2
2
hyperbolay
x
b
ayxb
y
a
x
11
0
01
)(1
2
2
2
2
2
2
0010
00)(22
y
x
y
xyxyxorxy
Goal : Try to transform the quadratic equation
into standard form by suitable translation
and rotation
Example : (No xy term)
The eigenvalues of the quadratic terms are 9 , 4
→ ellipse
→
→
011164189 22 yyxx
362419 22 yx
1
3
2
2
12
2
2
2
yx
)21(
Example : (Have xy term)
→
→ By direct computation
is orthogonal
( why does such U exist ? )
→ Let the original equation becomes
08323 22 yxyx
831
13)(
y
xyx
00
00
45cos45sin
45sin45cos
2
1
2
12
1
2
1
,40
02UUUA T
y
xU
y
x
840
02)(
y
xyx
x
y
x
y
045
Example :
→
→ Let
→
or
or
→
0428323 22 yyxyx
04)280(31
13)(
y
x
y
xyx
2
1
2
12
1
2
1
, Uy
xU
y
x
x
y
x
y
4)280(40
02)(
y
xU
y
xyx
48)(4)(2 22 yxyx
8)1(2)2( 22 yx
1&2,
8)(2)( 22
yyxx
yx
x
y
Optimization :
Let
It is known from Taylor’s Theorem of Calculus that
Where is Hessian matrix
is local extremum
If
then is a local minimum
2: CRRf n
...
)()())(()()( 00000
TOH
xxHxxxxxfxfxf T
nnxx RxfHji
))(( 0
))(,( 00 xfx 0)( 0 xf
)0.(
0)()(0&0)( 000
resp
xxHxxxf T
}0{)( 00* xxxxNx
))(,( 00 xfx .)max.(resp
Def : A real symmetric matrix A is said to be
(i) Positive definite denoted by
(ii) Negative definite denoted by
(iii) Positive semidefinite denoted by
(iv) Negative semidefinite denoted by
example :
is indefinite
question : Given a real symmetric matrix , how to determine
its definiteness efficiently ?
0,00 xAxxifA T
0,00 xAxxifA T
0,00 xAxxifA T
0,00 xAxxifA T
040
02
A
040
02
A
040
02
A
Theorem6.5.1: Let Then
Pf : let be eigenpair of A
Suppose
Let be an orthonormal eigen-basis of A
(Why can assume this ? )
""
""
nnT RAA RAA )(0
),(&0 xA 2
xxAxT
02 x
xAxT
RA)(}{ 1 nxx
0A
n
n
iii
n someforxXRx 11
,
)?(0
)()(
1
2
11
why
xxxAx
n
iii
n
iiii
Tn
iii
T
Example: Find local extrema of
Sol :
Thus f has local maximum at while
are saddle points
22 cosy3)( zxzxzyxf
)0,0,0()2siny,3,2( zxzxf
)0,,0()y,,( nzx
201
030
102
)0,2,0( nH
1,3,3)( H
201
030
102
)0,)12(,0( nH
1,3,3)( H)0,2,0( n
)0,)12(,0( n
Positive Definite Matrices :
Property I : P is nonsingular
Property II :
and all the leading principal submatrices of
A are positive definite
0& PRPP nnT
0det(P)& )det(P)( i
iPPRPP iinnT ,00&
Property III :
P can be reduced to upper triangular form
using only row operation III and the pivots
elements will all be positive
Sketch of the proof :
& determinant is invariant under row
operation of type III
Continue this process , the property can be proved
0& PRPP nnT
011 P
22
1211
2221
12112 0 P
PP
PP
PPP
02 P0)1(
22 P
Property IV : Let Then
(i) A can be decompose as A=LU where L is lower triangular & U is upper triangular
(ii) A can be decompose as A=LU where L is lower triangular & U is upper triangular with all the
diagonal element being equal to 1 , D is an
diagonal matrix
Pf : by Gaussian elimination and the fact that the
product of two lower (upper) triangular matrix
is lower (upper) triangular
0& ARAA nnT
Example:
Thus A=LU
Also A=LDU with
522
2102
224
A
430
390
2242
1
2
11213
AEE
UAE
300
390
2243
123
13
1
2
1
012
1000
3
1
2
1
2
1 where 231312 EEEL
1003
110
2
1
2
11
&
300
090
004
,
13
1
2
1
012
1001
UDL
Property V : Let
If
Pf :
LHS is lower triangular & RHS is upper triangular
with diagonal elements 1
0& ARAA nnT
212121
222111
&,
then,
UUDDLL
UDLUDLA
11211
12
12
UUDLLD
121
12 UUIUU
IDLLD 11
12
12
(why?) 11
21
1211
2 ILLDDLL
211
1221 DDIDDLL
Property VI : Let
Then A can be factored into
where D is a diagonal matrix & L is lower triangular
with 1’s along the diagonal
Pf :
Since the LDU representation is unique
0& ARAA nnT
TTAA DLULDU
LDUAT
Let
TLDLA
TLU
Property VII : (Cholesky decomposition)
Let
Then A can be factored into where
L is lower triangular with positive diagonal
Hint :
TT LDLDLDLA ))(( 2
1
2
1
0& ARAA nnT
TLLA
Example: We have seen that
Note that
Define we have the
Choleoky decomposition
LUA
300
390
224
13
1
2
1
012
1001
522
2102
224
LDUA
1003
110
2
1
2
11
300
090
004
13
1
2
1
012
1001
Also
TTT LDLALUAA
LDL 2
1
1
TLLA 11
Theorem6.6.1 : Let , Then the followings are equivalent:
(i) A>0
(ii) All the leading principal submatrices have positive determinants.
(iii) A ~ U only using elementary row operation of type III. And the pivots are all positive , where U is an upper triangular matrix.
(iv) A has Cholesky decomposition LLT.
(v) A can be factored into BTB for some nonsingular matrix B
nnT RAA
row
Pf : We have shown that (i) (ii) (iii) (iv) In addition , (iv) (v) (i) is trivial
Housholder Transformation :
Def : Then the matrix
is called
Housholder transformation
Geometrical lnterpretation:
Q is symmetric ,
Q is orthogonal ,
What is the eigenvalues , eigenvectors
and determinant of Q ?
1Let 2u
TuuIQ 2
QQT 1, QQIQQ TT
Given , Find
x u
1)2( exuuIxH T
2221 xxHe
1
1
ex
exu
QR factorization
xu
1e
....
ˆ0
01,
*0
*0
*...
*0
*......
222
21
HHHH
HHAHH
k
kk
Theorem : Let and be a
SVD of A with
Then
Pf :
Cor : Let be nonsingular with
singular values
Then and
nnRA TVUA )( 1 ndiag
)(max12AAA T
12
max
12
max
1
2
max
12
max
12
yxVyxV
xVUxAA
y
TT
x
T
xx
nnRA
n 1
ii 0n
A1
2
1
Application : In solving What is the effect of the
Solution when present measurement error ?
bxA b
bxAbb~~ and ~
~Let
)1(~
.~ 1 bbAxx
b
bbAA
x
xx
A
bx
xAxAb
~
..~
)2(
. , Also
1)2)(1(
is said to be the condition number of A
If A is orthogonal then
This means that , due to the error in b
the deviation of the associated solution of
is minimum if A is orthogonal
1.)( AAAK
nn
n
RAAK , 1)( 1
1)( AK
bAx
Example :
A is close to singular
Note that , is the solution for and is the solution for
What does this mean ?
Similarly , i.e. small deviation in x results in large deviation in b This is the reason why we use orthogonal factorization in Numerical solving Ax=b
11
001.11A
1.)(& AAAK
1
1x
001.2
001.2b
0
2~x
2
2~b
1~
but22
001.0~
x
xx
b
bb
1
0~ ,
0
1
1001
1001~ , 1000
1000bbxx