Post on 29-Mar-2020
LA
Linear Algebra
Chih-Wei Yi
Dept. of Computer ScienceNational Chiao Tung University
January 6, 2010
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
Eigenvalues and Eigenvectors
De�nition
Let A be an n� n matrix. A scalar λ is called an eigenvalues of Aif there exist a nonzero vector x such that Ax = λx. The vector xis called an eigenvector belonging to λ. The eigenspacecorresponding to λ is the collection of all eigenvectors belonging toλ and the zero vector 0.
Theorem
If λ is an eigenvalues of a matrix A, the eigenspace correspondingto λ is a vector subspace.
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
How to Find Eigenvalues and Eigenvectors
If Ax = λx, we have (A� λI) x = 0. Here x is a nonzerovector, so det (A� λI) = 0. Thus,
The n-degree polynomial pA (λ) = det (A� λI) is called thecharacteristic polynomial for the matrix A, and the equationdet (A� λI) = 0 is called the characteristic equation for thematrix A.λ is an eigenvalue of A if and only if λ is a root of thecharacteristic equation.If λ is an eigenvalue of A, the eigenspace corresponding to λ isthe solution of an n� n homogeneous system (A� λI) x = 0.
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
Example
Let A =
24 3 �1 �22 0 �22 �1 �1
35. Find the eigenvalues of A and thecorresponding eigenvspaces.
Solution
First, �nd the characteristic polynomial.
pA (λ) = det (A� λI) =
������3� λ �1 �22 �λ �22 �1 �1� λ
������=
0@ (3� λ) (�λ) (�1� λ) + (�1) (�2) (2)+ (�2) (2) (�1)� (�2) (�λ) (2)
� (�1) (2) (�1� λ)� (3� λ) (�2) (�1)
1A= �λ (λ� 1)2 .
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
(Cont.)
Let λ1 = 1. Then, A� λ1I =
24 2 �1 �22 �1 �22 �1 �2
35. Solve(A� λ1I) x = 0.24 2 �1 �2
2 �1 �22 �1 �2
������000
35!24 1 � 1
2 �10 0 00 0 0
������000
35 .Let x2 = s and x3 = t. We have
N (A� λ1I) =
8<:24 1
2 s + tst
35 : s, t 2 R
9=; .So, for λ = 1, the eigenspace is span
�(1, 2, 0)T , (1, 0, 1)T
�.
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
(Cont.)
Let λ2 = 0. Then, A� λ2I =
24 3 �1 �22 0 �22 �1 �1
35. Solve(A� λ2I) x = 0.24 3 �1 �2
2 0 �22 �1 �1
������000
35!24 1 0 �10 1 �10 0 0
������000
35 .Let x3 = s. We have
N (A� λ2I) =
8<:24 sss
35 : s, t 2 R
9=; .So, for λ = 0, the eigenspace is span
�(1, 1, 1)T
�.
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
Example
Let A =
24 2 �3 11 �2 11 �3 2
35. Find the eigenvalues of A and thecorresponding eigenvspaces.
Solution
The characteristic polynomial of A is
pA (λ) = det (A� λI) =
������3� λ �1 �22 �λ �22 �1 �1� λ
������= �λ (λ� 1)2 .
1 Let λ1 = 0. The eigenspace is span�(1, 1, 1)T
�.
2 Let λ2 = 1. The eigenspace is span�(3, 1, 0)T , (�1, 0, 1)T
�.
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
Some Properties
Theorem
Let A be an n� n matrix, and λ is a scalar. The followingstatements are equivalent:
1 λ is an eigenvalue of A.2 (A� λI) x = 0 has a nontrivial solution.3 N (A� λI) 6= f0g.4 A� λI is singular.5 det (A� λI) = 0.
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
Independency of Eigenvectors
Theorem
If λ1,λ2, � � � ,λk are distinct eigenvalues of an n� n matrix Awith corresponding eigenvectors x1, x2, � � � , xk , then x1, x2, � � � , xkare linearly independent.
Proof.
First, show fx1, x2g is linearly independent. Then, assumefx1, x2, � � � , xlg is linearly independent, and provefx1, x2, � � � , xl+1g is also linear independent.
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
Determinant and Trace
De�nition (Trace)
If A is an n� n matrix, the trace of A, denoted as tr (A), isa11 + a22 + � � �+ ann.
Theorem
If an n� n matrix A is with n distinct eigenvalues λ1,λ2, � � � ,λn,then
1 det (A) = λ1 � λ2 � � � � � λn.
2 tr (A) = λ1 + λ2 + � � �+ λn.
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
Proof.
pA (λ) = det (A� λI) =
���������a11 � λ a12 � � � a1na21 a22 � λ � � � a21...
.... . .
...an1 an2 � � � ann � λ
���������= (a11 � λ) det (M11) +
n
∑i=2ai1 (�1)i+1 det (Mi1)
= (�1)n (λ� λ1) (λ� λ2) � � � (λ� λn) .
LA
Eigenvalues
Section 1 Eigenvalues and Eigenvectors
Similar Matrices
Theorem
Let A and B be n� n matrices. If B is similar to A, then the twomatrices both have the same characteristic polynomial andconsequently both have the same eigenvalues.
Proof.
It is easy to verify.
pB (λ) = det (B� λI)= det
�S�1AS� λI
�= det
�S�1 (A� λI)S
�= det
�S�1
�det (A� λI) det (S)
= det (A� λI)= pA (λ) .
LA
Eigenvalues
Section 2 Systems of Linear Di¤erential Equations
Section 2 Systems of Linear Di¤erentialEquations
NOT COVERED IN THIS CLASS
LA
Eigenvalues
Section 3 Diagonalization
Diagonalization
De�nition
An n� n matrix A is said to be diagonalizable if there exist anonsigular matrix X and a diagonal matrix D such thatX�1AX = D. We say that X diagonalized A.
Theorem
An n� n matrix A is diagonalizable if and only if A has n linearlyindependent eigenvectors.
LA
Eigenvalues
Section 3 Diagonalization
Example
Assume A =
24 3 �1 �22 0 �22 �1 �1
35. We havepA (λ) = det (A� λI) = �λ (λ� 1)2.For λ1 = 0, we have a eigenvector (1, 1, 2)
T .For λ2 = 1, we have eigenvectors (1, 2, 0)
T and (1, 0, 1)T .
Let X =
24 1 1 11 2 01 0 1
35. We have X�1 =24 �2 1 2
1 0 �12 �1 �1
35 andD = X�1AX.
LA
Eigenvalues
Section 3 Diagonalization
Defective Matrices
De�nition
An n� n matrix A has fewer than n linearly independenteigenvectors, we say that A is defective. An defective matrix is notdiagonalizable.
Examples
A =
24 2 0 00 4 01 0 2
35 and B =24 2 0 0�1 4 0�3 6 2
35.
LA
Eigenvalues
Section 3 Diagonalization
Polynomials of Matrices
If p (x) = amxm + am�1xm�1 + � � �+ a0 is a m-degreepolynomial and A is an n� n matrix, then
p (A) = amAm + am�1Am�1 + . . .+ a0I.
If A is diagonalizable by X, i.e. D = X�1AX or A = XDX�1,then
p (A) = amAm + am�1Am�1 + . . .+ a0I= X(amDm + am�1Dm�1 + . . .+ a0I)X�1
= Xp (D)X�1.
LA
Eigenvalues
Section 3 Diagonalization
Polynomials of Matrices (Cont.)
In addition, if D =
26664λ1 0 � � � 00 λ2 � � � 0...
.... . .
...0 0 � � � λn
37775, then
p (D) =
26664p (λ1) 0 � � � 00 p (λ2) � � � 0...
.... . .
...0 0 � � � p (λn)
37775.
LA
Eigenvalues
Section 3 Diagonalization
Example
Assume p (x) = x2 + 3x � 2. Compute p (A) for
A =��2 �61 3
�.
LA
Eigenvalues
Section 3 Diagonalization
Solution
The eigenvectors are x1 = (�2, 1)T for λ1 = 1 and x2 = (�3, 1)T
for λ2 = 0. Let X =��2 �31 1
�. Then,
A = XDX�1 =��2 �31 1
� �1 00 0
� �1 31 1
�,
and
p (A) = Xp (D)X�1 =��2 �31 1
� �p (1) 00 p (0)
� �1 31 1
�=
��2 �31 1
� �2 00 �2
� �1 31 1
�=
�2 �60 4
�.
LA
Eigenvalues
Section 3 Diagonalization
Exponentials of Matrices
According to Taylor�s series, we have
ex = 1+ x +12!x2 + � � � .
For any n� n matrix A , eA is given by
eA = I+A+12!A2 + � � � .
If A is diagonalizable by X, i.e. D = X�1AX or A = XDX�1,then eA = XeDX�1.
LA
Eigenvalues
Section 3 Diagonalization
Exponentials of Matrices (Cont.)
In addition, if D =
26664λ1 0 � � � 00 λ2 � � � 0...
.... . .
...0 0 � � � λn
37775, then
eD =
26664eλ1 0 � � � 00 eλ2 � � � 0...
.... . .
...0 0 � � � eλn
37775.
LA
Eigenvalues
Section 3 Diagonalization
Example
Compute eA for A =��2 �61 3
�.
Solution
The eigenvectors are x1 = (�2, 1)T for λ1 = 1 and x2 = (�3, 1)T
for λ2 = 0. Let X =��2 �31 1
�. Then,
eA =
��2 �31 1
� �e1 00 e0
� �1 31 1
�=
�3� 2e 6� 6ee � 1 3e � 2
�.
LA
Eigenvalues
Section 3 Diagonalization
Markov Chains
De�nition
A stochastic process is any sequence of experiments for which theoutcome at any stage depends on chance. A Markov process is astochastic process with the following properties:
The set of possible outcomes or states is �nite.
The probability of the next outcome depends only on theprevious outcome.
The probabilities are constant over time.
De�nition
A vector is called a probability vector if all its entries arenonnegative and the sum of its entries is 1.
LA
Eigenvalues
Section 3 Diagonalization
Blue-Green Color Blindness (Sex-Linked Genes)
Let x (i )1 be the proportion of genes for color blindness in the male
population, and x (i )2 be the proportion of genes for color blindnessin the female population. Then,(
x (1)1 = x (0)2x (1)2 = 1
2x(0)1 + 1
2x(0)2
=)�0 112
12
� "x (0)1x (0)2
#=
"x (1)1x (1)2
#.
Let A =�0 112
12
�. We have
A =�1 �21 1
� �1 00 � 1
2
� � 13
23
� 13
13
�.
LA
Eigenvalues
Section 3 Diagonalization
(Cont.)
limn!∞
x(n) = limn!∞
Anx(0)
= limn!∞
�1 �21 1
� �1 00 � 1
2
�n � 13
23
� 13
13
�x(0)
= limn!∞
�1 �21 1
� �1n 00
�� 12
�n �n � 13
23
� 13
13
�x(0)
=
24 x (0)1 +2x (0)23
x (0)1 +2x (0)23
35 .
LA
Eigenvalues
Section 3 Diagonalization
Theorems of Markov Chains
Theorem
If a Markov chain with an n� n transition matrix A converges to asteady-state vector x, then
1 x is a probability vector.2 λ1 = 1 is an eigenvalue of A and x is an eigenvectorbelonging to λ1 = 1.
Theorem
If λ1 = 1 is a dominant eigenvalue1 of a stochastic matrix A, thenthe Markov chain with transition A will converge to a steady-statevector.
1An eigenvalue λ of a matrix A is called a dominant eigenvalue if��λ0�� < jλj
for all other eigenvalues λ0 of A.
LA
Eigenvalues
Section 4 Hermitian Matrices
Complex Inner Products
De�nition
Let V be a vector space over the complex numbers. An innerproduct on V is an operation that assigns to each pair of vectors zand w in V a complex number hz,wi satisfying the followingconditions.
hz, zi � 0. hz, zi = 0 if and only if z = 0.hz,wi = hw, zi for all z and w in V.hαz+ βw,ui = α hz,ui+ β hw,ui.
Problem
Prove that hu, αz+ βwi = α hu, zi+ β hu,wi.
LA
Eigenvalues
Section 4 Hermitian Matrices
Lengths of Complex Numbers and Complex Vectors
If α = a+ bi is a complex scalar, the length of α is given by
jαj =p
αα =pa2 + b2.
If z = (z1, z2, . . . , zn)T is a vector in Cn, the length of z is
given by
kzk =�jz1j2 + jz2j2 + � � �+ jzn j2
�1/2
= (z1z1 + z2z2 + � � �+ znzn)1/2
=�zT z
�1/2
Let zH = zT . Then, kzk =�zHz
�1/2.
LA
Eigenvalues
Section 4 Hermitian Matrices
An Complex Inner Product
De�nition
hz,wi = wHz for all z and w in Cn.
Example
If z =�5+ i1� 3i
�and w =
�2+ i�2+ 3i
�, then
hz,wi = wHz =�2� i �2� 3i
� � 5+ i1� 3i
�= (2� i) (5+ i) + (�2� 3i) (1� 3i) = 0
hz, zi = zHz =�5� i 1+ 3i
� � 5+ i1� 3i
�= 36
hw,wi = wHw =�2� i �2� 3i
� � 2+ i�2+ 3i
�= 18
LA
Eigenvalues
Section 4 Hermitian Matrices
Conjugate and Hermitian
De�nition
M is a m� n matrix with mij = aij + ibij for each i and j . IfA = (aij ) and B = (bij ), then
M = A+ iB.The conjugate of M, denoted by M, is A� iB.MH is the transport of M.
Theorem
If A,B 2 Cm�n and C 2 Cn�r , then�AH�H= A.
(αA+ βB)H = αAH + βBH .
(AC)H = CHAH .
LA
Eigenvalues
Section 4 Hermitian Matrices
Hermitian Matrices
De�nition
A matrix M is said to be Hermitian if M =MH .
Example
M =
�3 2� i
2+ i 4
�.
Theorem
The eigenvalues of a Hermitian matrix are all real. Furthermore,eigenvectors belonging to distinct eigenvalues are orthogonal.
LA
Eigenvalues
Section 4 Hermitian Matrices
Proof.
Let A be a Hermitian matrix, λ be an eigenvalue of A, and x bean eigenvectors belonging to λ.
Let α = xHAx. α is real sinceα = αH =
�xHAx
�H= xHAx = α.
In addition, since we have α = xHAx = xHλx = λ kxk2, wehave λ = α
kxk2 is real.
If x1 and x2 are eigenvectors belonging to distinct eigenvalues λ1and λ2,respectively.
(Ax1)H x2 = xH1 A
Hx2 = xH1 Ax2 = λ2xH1 x2.
(Ax1)H x2 =
�xH2 Ax1
�H=�λ1xH2 x1
�H= λ1xH1 x2.
It follows that hx2, x1i = xH1 x2 = 0.
LA
Eigenvalues
Section 4 Hermitian Matrices
Unitary
De�nition
An n� n matrix U is said to be unitary if its column vectors forman orthonormal set in Cn, i.e. UHU = I and U�1 = UH .
PS. A real unitary matrix is an orthogonal matrix.
Corollary
If the eigenvalues of a Hermitian matrix A are distrinct, then thereexists a unitary matrix U that diagonalizes A.
LA
Eigenvalues
Section 4 Hermitian Matrices
Example
Assume A =�
2 1� i1+ i 1
�. Then,
pA (λ) =
���� 2� λ 1� i1+ i 1� λ
���� = λ (λ� 3).
λ1 = 3) x1 =�1� i1
�; λ2 = 0) x2 =
��11+ i
�.
u1 = 1kx1k =
1p3
�1� i1
�; u2 = 1
kx2k =1p3
��11+ i
�.
Let U = 1p3
�1� i �11 1+ i
�. Then,
UHAU =1p3
�1+ i 1�1 1� i
� �2 1� i
1+ i 1
� �1� i �11 1+ i
�=
�3 00 0
�.
LA
Eigenvalues
Section 4 Hermitian Matrices
Schur�s Theorem
Theorem
For each n� n matrix A, there exists a unitary matrix U such thatUHAU is upper triangular.
The factorization A = UTUH is referred to as the Schurdecomposition of A.If A is Hermitian, the matrix T will be diagonal. (SpectralTheorem)
LA
Eigenvalues
Section 4 Hermitian Matrices
Proof of Schur�s Theorem
Proof.
This theorem can be proved by induction on n.If n = 1, obviously the result is correct.Assume the hypothesis holds for k � k matrices, and let A be a(k + 1)� (k + 1) matrix.
Let λ1 be an eigenvalue of A, and let w1 be a uniteigenvector belonging to λ1. Using the Gram-Schmidtprocess, construct an orthonormal basis fw1,w2, � � � ,wk+1g.
WHAW is a matrix of the form
26664λ1 � � � � �0... Mk�k0
37775.
LA
Eigenvalues
Section 4 Hermitian Matrices
(Cont.)
Since M is a k � k matrix, there exist a k � k unitary matrix V1such that VH1 MV1 = T1, where T1 is triangular. Let
V =
266641 0 � � � 00... V10
37775 .Here V is unitary and
VHWHAWV
=
26664λ1 � � � � �0... VH1 MV10
37775 =26664
λ1 � � � � �0... T10
37775 = T.
LA
Eigenvalues
Section 4 Hermitian Matrices
(Cont.)
Let U =WV. The matrix U is unitary, since
UHU = (WV)HWV = VHWHWV = I
andUHAU = T.
LA
Eigenvalues
Section 4 Hermitian Matrices
Example
Let A =
24 4625 �1 28
250 3 0325 2 29
25
35. Find the Schur decomposition of A.Solution
First, �nd an eigenvalue and an eigenvector for the eigenvalue.
pA (λ) = det (A� λI)
= det
0@24 4625 � λ �1 28
250 3� λ 0325 2 29
25 � λ
351A= �λ3 + 6λ2 � 11λ+ 6
= � (λ� 1) (λ� 2) (λ� 3) .
LA
Eigenvalues
Section 4 Hermitian Matrices
Solution (Cont.)
Let λ = 1 and solve (A� λI)X = 0. Then,24 2125 �1 28
250 2 0325 2 4
25
������000
35!24 1 0 4
30 1 00 0 0
������000
35
and we get an eigenvector
24 � 4301
35 for λ = 1. It can be veri�ed
that
8<:24 � 4
301
35 ,24 010
35 ,24 001
359=; is a basis of R3.
LA
Eigenvalues
Section 4 Hermitian Matrices
Solution (Cont.)
Applying Gram-Schmidt process, we have w1 =
24 � 45035
35,w2 =
24 010
35, and w3 =24 3
5045
35. Let W =
24 � 45 0 3
50 1 035 0 4
5
35. Then,
WHAW =
24 � 45 0 3
50 1 035 0 4
5
35H 24 4625 �1 28
250 3 0325 2 29
25
3524 � 45 0 3
50 1 035 0 4
5
35=
24 1 2 �10 3 00 1 2
35 .
LA
Eigenvalues
Section 4 Hermitian Matrices
Solution (Cont.)
Now, consider M =
�3 01 2
�. Then,
�01
�is an eigenvector for
the eigenvalue λ = 2. Let V1 =�0 11 0
�. We have
VH1 AV1 =�0 11 0
� �3 01 2
� �0 11 0
�=
�2 10 3
�.
So, Let V =
24 1 0 00 0 10 1 0
35.
LA
Eigenvalues
Section 4 Hermitian Matrices
Solution (Cont.)
We have
VH�WHAW
�V =
24 1 0 00 0 10 1 0
3524 1 2 �10 3 00 1 2
3524 1 0 00 0 10 1 0
35=
24 1 �1 20 2 10 0 3
35 .Let
U =WV =
24 � 45 0 3
50 1 035 0 4
5
3524 1 0 00 0 10 1 0
35 =24 � 4
535 0
0 0 135
45 0
35 .
LA
Eigenvalues
Section 4 Hermitian Matrices
Solution (Cont.)
We have
UHAU =
24 � 45 0 3
535 0 4
50 1 0
3524 4625 �1 28
250 3 0325 2 29
25
3524 � 45
35 0
0 0 135
45 0
35=
24 1 �1 20 2 10 0 3
35and
A =
24 � 45
35 0
0 0 135
45 0
3524 1 �1 20 2 10 0 3
3524 � 45 0 3
535 0 4
50 1 0
35 .
LA
Eigenvalues
Section 4 Hermitian Matrices
Spectral Theorem
Theorem
If A is Hermitian, then there exists a unitary matrix U thatdiagonalizes A.
Proof.
There is a unitary matrix U such that T = UHAU, where T isupper triangular. TH = (UHAU)H = UHAHU = UHAU = T.Therefore, T is Hermitian and diagonal.
Corollary
If A is a real symmetric matrix, then there exists an orthogonalmatrix U that diagonalizes A.
LA
Eigenvalues
Section 4 Hermitian Matrices
Example
A =
24 0 2 �12 3 �2�1 �2 0
35.Solution
pA (λ) = �λ3 + 3λ2 + 9λ+ 5 = (1+ λ)2 (5� λ).λ1 = �1,λ2 = �1) x1 = (1, 0, 1)
T , x2 = (�2, 1, 0)T .λ3 = 5) x3 = (�1,�2, 1)T .Applying the Gram-Schmidt process, we have
w1 = (1, 0, 1)T and u1 = 1
kw1kw1 =�1p2, 0, 1p
2
�Tw2 = (�1, 1, 1)T and u2 = 1
kw2kw2 =�� 1p
3, 1p
3, 1p
3
�Tw3 = x3 = (�1,�2, 1)T and u2 = 1
kw3kw3 =�� 1p
6,� 2p
6, 1p
6
�T.
LA
Eigenvalues
Section 4 Hermitian Matrices
Solution (Cont.)
Let U =
2641p2� 1p
3� 1p
60 1p
3� 2p
61p2
1p3
1p6
375.Then, UHAU =
24 �1 0 00 �1 00 0 5
35.
LA
Eigenvalues
Section 4 Hermitian Matrices
Normal Matrices
Some non-Hermitian matrices possess complete orthonormalsets of eigenvectors.
e.g. skew symmetric and skew Hermitian matrices. (A is skewHermitian if AH = �A).
If A is a matrix with a complete orthonormal set ofeigenvectors, A = UDUH , where U is unitary and D is adiagonal matrix.
In general, DH 6= D.AAH = AHA.
AAH = UDUHUDHUH = UDDHUH .AHA = UDHUHUDUH = UDDHUH .
LA
Eigenvalues
Section 4 Hermitian Matrices
Normal Matrices
De�nition
A matrix A is said to be normal if AAH = AHA.
Theorem
A matrix A is normal if and only if A possesses a completeorthonormal set of eigenvectors.
LA
Eigenvalues
Section 4 Hermitian Matrices
Proof.
There exists a unitary matrix U and a triangular matrix T suchthat T = UHAU.We have TTH = UHAAHU and THT = UHAHAU. SinceAAH = AHA, T is also normal.Comparing the diagonal elements of TTH and THT, we have
jt11j2 + jt12j2 + � � �+ jt1n j2 = jt11j2
jt22j2 + � � �+ jt2n j2 = jt12j2 + jt22j2...
jtnn j2 = jt1n j2 + jt2n j2 + � � �+ jtnn j2
It follows that tij = 0 whenever i 6= j . Thus, U diagonalize A andthe column vectors of U are eigenvectors of A.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Section 5 The Singular Value Decomposition
LA
Eigenvalues
Section 5 The Singular Value Decomposition
SUV Decomposition
A = UΣVT
A: an m� n matrix. (Assume m � n.)U: an m�m orthogonal matrix.
V: an n� n orthogonal matrix.
Σ =
266666664
σ1 0 0 0
0 σ2 0...
0 0. . . 0
...... σn
0 0 � � �
377777775: an m� n matrix whose o¤
diagonal entries are all 0�s and whose diagonal elementssatisfy σ1 � σ2 � � � � � σn � 0.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
The SVD Theorem
Objectives
The σi�s determined by this factorization are unique and calledthe singular value decomposition.The rank of A equals the number of nonzero singular values.
Theorem
If A is an m� n matrix, then A has a singular value decomposition.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Proof.
If A = UΣVT , then ATA = VΣTΣVT is diagonalized by V.
Let λ1,λ2, � � � ,λn be eigenvectors of ATA sorted indescending order, i.e. λ1 � λ2 � � � � � λn � 0, andv1, v2, � � � , vn are corresponding eigenvectors. (Actually, it istrue that ATA can be diagonalized by a unitary matrix.)We have λ1 � 0 for all i = 1, 2, � � � , n. (Hint:kAvik2 = λi kvik2.)Then, V = (v1, � � � , vn) and σi = λ1/2
i .
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Cont.
Assume A has rank r , then ATA also has rank r .
λ1 � λ2 � � � � � λr > 0 and λr+1 = λr+2 = � � � = λn = 0.
σ1 � σ2 � � � � � σr > 0 and σr+1 = σr+2 = � � � = σn = 0.
For i = r + 1, � � � , n, since λi = 0, we have ATAvi = 0.fvr , vr+1, � � � , vng is an orthonormal basis forN�ATA
�= N (A).
Let V1 = fv1, v2, � � � , vrg, V2 = fvr+1, vr+2, � � � , vng, and
Σ1 =
264 σ1. . .
σr
375.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Cont.
Assume A = UΣVT . Then, AV = UΣ.
Comparing the �rst r column of each side, we see thatAvj = σjuj for j = 1, � � � , r .Let uj = 1
jσj jAvj for j = 1, � � � , r , and U1 = (u1, � � � ,ur ).We have AV1 = U1Σ1.The column vector of U1 form an orthonormal set. (why?)
Since rank (A) = r , the column vector of U1 form anorthonormal set of R (A).
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Cont.
The vector space R (A)? = N�AT�has dimension m� r .
Let fur+1,ur+2, � � � ,ung be an orthonormal basis forN�AT�, and set U2 = (ur+1,ur+2, � � � ,un) and
U = (U1 U2).Then,
UΣVT = (U1 U2)�
Σ1 00 0
��VT1VT2
�= U1Σ1VT1 = AV1V
T1 = A.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Observations
The singular value σ1, σ2, � � � , σn of A are unique; however,the matrices U and V are not unique.Since V diagonalizes ATA, vj�s are eigenvectors of ATA.Since AAT = UΣΣTU, it follows that U diagonalizes AAT
and that the uj�s are eigenvectors of AAT .Comparing the jth columns of each side of the equationAV = UΣ, we get Avj = σjuj for j = 1, � � � , n. Similarly,from ATU = VΣT , we have Auj = σjvj for j = 1, � � � , n andAT uj = 0 for j = n+ 1, � � � ,m.
The vj�s are called the right singular vectors of A.The uj�s are called the left singular vectors of A.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Observations (cont.)
If A has rank r , then
v1, v2, � � � , vr form an orthonormal basis for R�AT�.
vr+1, vr+2, � � � , vn form an orthonormal basis for N (A).u1, � � � ,ur form an orthonormal basis for R (A).ur+1,ur+2, � � � ,un form an orthonormal basis for N
�AT�.
The rank of the matrix A is equal to the number of itsnonzero singular values (where singular values are countedaccording to multiplicity).
In the case that A has rank r < n, if we setU1 = (u1, � � � ,ur ),V1 = fv1, v2, � � � , vrg, and Σ1 as inequation (1), then A = U1Σ1VT1 . The factorization is calledthe compact form of the SVD of A.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Example
Compute the SVD of A =
24 1 11 10 0
35.Solution
First, ATA =�2 22 2
�has an eigenvector (1, 1)T for eigenvalue
λ1 = 4, and an eigenvector (1,�1)T for eigenvalue λ2 = 0. So thesingular values of A are σ1 =
p4 = 2 and σ2 = 0. So, the singular
values of A are σ1 = 2 and σ2 = 0, and V = 1p2
�1 11 �1
�.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Solution ((Cont.))
From observation 4,
u1 =1
σ1Av1 =
12
24 1 11 10 0
35 " 1p21p2
#=
2641p21p20
375 .The remaining column vectors of U must from an orthonormalbasis for N
�AT�.nw2 = (1,�1, 0)T ,w3 = (0, 0, 1)T
ois a basis of N
�AT�.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Solution (Cont.)
Applying the Gram-Schmidt process,(u2 =
�1p2,� 1p
2, 0�T
,u3 = (0, 0, 1)T
)
is an orthonormal basis of N�AT�. So, U =
2641p2
1p2
01p2� 1p
20
0 0 1
375,and
A = UΣVT =
2641p2
1p2
01p2� 1p
20
0 0 1
37524 2 00 00 0
35 " 1p2
1p2
1p2� 1p
2
#.
LA
Eigenvalues
Section 5 The Singular Value Decomposition
The Closest Matrix
If A is an m� n matrix of rank r and 0 < k < r , we wouldlike to �nd a matrix in Rm�n of rank k that is closest to Aw.r.t. the Frobenius norm.
In other words, letM be the set of all m� n matrices of rankk or less. I would like to �nd a matrix X inM such that
kA�XkF = minS2M
kA� SkF .
Lemma
If A is an m� n matrix and Q is an m�m orthogormal matrix,then
kQAkF = kAkF .
LA
Eigenvalues
Section 5 The Singular Value Decomposition
Theorem
Let A = UΣVT be an m� n matrix, and letM denote the set ofall m� n matrices of rank k or less, where 0 < k < rank (A). If Xis a matrix inM satisfying kA�XkF = minS2M kA� SkF , then
kA�XkF =�σ2k+1 + σ2k+2 + � � �+ σ2r
�1/2.
In particular, if A0= UΣ0VT , where
Σ0 =
26664σ1
. . . 0σk
0 0
37775 =�
Σk 00 0
�
then A�A0 F = �σ2k+1 + σ2k+2 + � � �+ σ2r�1/2
= minS2M
kA� SkF .