1
Linear Algebra
Chi-Min Liu
Sep. 11, 2010Department of Computer Science and Information [email protected]
2
Preface
Time: EC122,
Wednesday 10:10 – 12:30.
Friday 3:30 – 4:30 (hw & test)
Textbooks
Score:
3 Exam (70%)
HW + MATLAB + Test (30%)
Linear Algebra with Applications
8th Edition
by
Steven J. Leon
3
Contents
Chapter 1: Matrices and Systems of Equations
Chapter 2: Determinants
Chapter 3: Vector Spaces
Chapter 4: Linear Transformations
Chapter 5: Orthogonality
Chapter 6: Eigenvalues
Chapter 7: Numerical Linear Algebra
Chapter 8: Iterative Mathods
Chapter 9: Canonical Forms
http://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869031-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869122-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869177-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869286-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869341-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869341-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869468-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869468-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869613-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869740-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869740-,00.htmlhttp://wps.prenhall.com/am_leon_linearalg_7/0,9761,2869744-,00.html
Chapter 1
Matrices and Systems of Equations
Systems of Linear Equations
Row Echelon Form
Matrix Arithmetic
Matrix Algebra
Elementary Matrices
Partitioned Matrices
4
5
1. Systems of Linear Equations
A linear system of m equations and n unknowns
6
1. Systems of Linear Equations
Example
7
1. Systems of Linear Equations
8
1. Systems of Linear Equations
Equivalent Systems
Definition
Two systems of equations involving the same variables are
said to be equivalent if they have the same solution set.
9
1. Systems of Linear Equations
Three Operations Types
Obtain an equivalent system easier to solve.
Definition-- Strict Triangular Form
A system is said to be in strict triangular form if in the kth
equation the coefficients of the first k-1 variables are all
zero and the coefficient of xk is nonzero (k=1, …, n)
10
1. Systems of Linear Equations
11
1. Systems of Linear Equations
12
2. Row Echelon Form
13
2. Row Echelon Form
14
2. Row Echelon Form
Definition–A linear system is said to be
overdetermined if there are more
equations than unknowns.
•A system of m linear equations in
n unknowns is said to be
underdetermined if there are fewer
equations than unknown.
http://www.math.drexel.edu/~pg/fb//java/la_applets/GaussElim/index.html
http://www.math.drexel.edu/~pg/fb/java/la_applets/GaussElim/index.html
15
2. Row Echelon Form
16
2. Row Echelon Form
17
2. Row Echelon Form
The process of using elementary row
operations to transform a matrix into reduced
row echelon form is called Gauss-Jordan
reduction.
2. Row Echelon Form
18
Using Gauss-Jordan reduction to solve the system
Sol: 022
03
03
4321
4321
4321
xxxx
xxxx
xxxx
19
Theorem 1.2.1
An m×n homogeneous system of linear equations has a nontrivial solution if n > m (underdetermined).
Pf:
A homogeneous system is always consistent. There are at most m lead variables. Since there are n variables altogether and n>m, there must be some free variables and they can be assigned arbitrary values. So, we can get a nontrivial solution if n>m.
20
3. Matrix Arithmetic
)( ijaA
21
3. Matrix Arithmetic
22
3. Matrix Arithmetic
23
3. Matrix Arithmetic
24
4. Matrix Algebra
25
Proof of Rule 4
is a m×n matrix, and and
are both n×r matrices. Let and
and
But
so that and hence A(B + C) = AB + AC.
)( ijaA )( ijbB )( ijcC
)( CBAD ACABE
n
k
kjkjikij cbad1
)(
n
k
kjik
n
k
kjikij cabae11
n
k
kjik
n
k
kjik
n
k
kjkjik cabacba111
)(
ijij ed
26
Proof of Rule 3
A be a m×n matrix, and B an n×r matrix and C an r×s matrix. Let and
The (i, j) entry of DC is
and the (i, j) entry of AE is
ABD BCE
r
l
ljklkj
n
k
klikil cbebad11
and
r
l
lj
n
k
klik
r
l
ljil cbacd1 11
n
k
lj
r
l
klik
n
k
kjik cbaea1 11
27
Example 1
12
01 and ,
23
12 ,
43
21CBA
1116
56
21
14
43
21)(BCA
1116
56
12
01
116
54)( CAB
28
Example 1
Thus
ThereforeA(B + C) = AB + AC
155
71
411
25
116
54
155
71
31
13
43
21)(
)(1116
56)(
ACAB
CBA
CABBCA
29
Example 1
Since
it follows that
(AB)C = DC = AE = A(BC)
n
k
r
l
ljklik
r
l
n
k
ljkliklj
r
l
n
k
klik cbacbacba1 11 11 1
30
Notation
If A is an n×n matrix and k is a positive integer, then
timesk
AAAAk
31
Example 2
If
then
In general
,11
11
A
22
22
11
11
11
112A
44
44
22
22
11
1123 AAAAAA
11
11
22
22nn
nn
nA
32
4. Matrix Algebra
33
Example
is a 3×3 identity matrix
100
010
001
)(
810
362
143
810
362
143
100
010
001
AIA
)(
810
362
143
100
010
001
810
362
143
AAI
34
Matrix Inversion
A real number a is said to have a multiplicative inverse if there exists a number b such that ab = 1.
Any nonzero number a has a multiplicative inverse b=1/a.
35
Definition
An n×n matrix A is said to be nonsingular orinvertible if there exists a matrix B such that AB = BA = I. The matrix B is said to be a multiplicative inverse of A.
• If B and C are both multiplicative inverse of A (i.e.,
BA = AB = I and CA = AC = I), then B = BI = B(AC)
= (BA)C = IC = C
• A matrix can have at most one multiplicative inverse.
• Notation: The inverse of A is denoted by A-1.
36
Example 3
The matrices and are inverse of each other, since
and
13
42
51
103
52
101
10
01
13
42
51
103
52
101
10
01
13
42
51
103
52
101
37
Example 4
The 3×3 matrices and
are inverse, since
and
100
410
321
100
410
521
100
010
001
100
410
521
100
410
321
100
010
001
100
410
321
100
410
521
38
Example 5
The matrix has no inverse.
Sol: Let
then
00
01A
2221
12111
bb
bbBA
10
01
0
0
00
01
21
11
2221
1211
b
b
bb
bbBA
39
4. Matrix Algebra
40
Note
If A1, A2,…, Ak are all nonsingular n×n matrices, then the product A1A2…Ak is nonsingular and
(A1A2…Ak)-1 = Ak
-1… A2-1A1
-1
41
4. Matrix Algebra
42
Example 6
Let
145
112
201
,
142
533
121
BA
9815
142334
5610
145
112
201
142
533
121
AB
9145
8236
153410
151
432
231
112
410
521TT AB
43
Definition
An n×n matrix A is said to be symmetric if AT = A (i.e., aij = aji)
354
513
432
A
354
513
432TA
• Example
44
5. Elementary Matrices
45
Example 1
100
001
010
1E
333132
232122
131112
333231
232221
131211
1
333231
131211
232221
333231
232221
131211
1
100
001
010
100
001
010
aaa
aaa
aaa
aaa
aaa
aaa
AE
aaa
aaa
aaa
aaa
aaa
aaa
AE
46
Example 2
300
010
001
2E
333231
232221
131211
333231
232221
131211
2
333231
232221
131211
333231
232221
131211
2
3
3
3
300
010
001
33300
010
001
aaa
aaa
aaa
aaa
aaa
aaa
AE
aaa
aaa
aaa
aaa
aaa
aaa
AE
47
Example 3
100
010
301
3E
33313231
23212221
13111211
3
333231
232221
331332123111
3
3
3
3:operationColumn
333:operation Row
aaaa
aaaa
aaaa
AE
aaa
aaa
aaaaaa
AE
48
Note
Suppose that E is an n×n elementary matrix. If A is an n×r matrix,premultiplying A by E has the effect of performing that same row operation on A. If B is an m×n matrix, postmultiplyingB by E is equivalent to performing that same column operation on B.
49
Theorem 1.5.1
If E is an elementary matrix, then E is nonsingular and E-1 is an elementary matrix of the same type.
• Proof
– Type I: interchange of two rows
E1E1 = I E1-1 = E1
50
Note
B is row equivalent to A if B can be obtained from A by a finite number of row operations.
Two augmented matrices (A | b) and (B| c) are row equivalent iff Ax = b and Bx = c are equivalent systems.
If A is row equivalent to B, B is row equivalent to A.
If A is row equivalent to B, and B is row equivalent to C, then A is row equivalent to C.
51
5. Elementary Matrices
Type II: multiplying the ith row of I by a nonzero scalar
rowth
1
1
1
1
2 i
0
0
E
52
5. Elementary Matrices
rowth
1
1
1
1
1
1
2 i
0
0
E
53
5. Elementary Matrices
Type III: adding m times the ith row to the jth row
rowth
rowth
1000
10
10
1
3
j
i
m
0
E
54
5. Elementary Matrices
rowth
rowth
1000
10
10
1
1
3
j
i
m
0
E
55
5. Elementary Matrices
56
Theorem 1.5.2
Equivalent conditions for Nonsingularity
Let A be an n×n matrix. The following are
equivalent:
(a) A is nonsingular.
(b) Ax = 0 has only the trivial solution 0.
(c) A is row equivalent to I.
• Pf: (a) (b)
If A is nonsingular (i.e., A-1 exists), then for Ax = 0
00xxxx 111 )ˆ(ˆ)(ˆˆ AAAAAI
57
5. Elementary Matrices
(b) (c)
If Ax = 0 has only the trivial solution 0
Ax = 0 Ux = 0 (using elementary rowoperation) where U is in row echelonform and U = Ek . . . E1A
From Theorem 1.2.1.
In U, the number of nonzero rows mustbe the same as unknowns (m = n)
58
5. Elementary Matrices
Thus,
U must be a strictly triangular matrix with diagonal elements all equal to 1.
∴ I will be the reduced row echelon form of A
∴ A is row equivalent to I
59
5. Elementary Matrices
(c) (a)
If A is row equivalent to I, there exist elementary matrices E1, E2, …, Ek such that
A = EkEk-1…E1I = EkEk-1…E1∵ E1, E2, …, Ek are all invertible
∴ A-1 = (EkEk-1…E1)-1 = E1
-1E2-1… Ek-1
-1 Ek-1
∴ A is nonsingular (invertible)
60
5. Elementary Matrices
(1) If A is nonsingular, and x is any solution to Ax=b, then we can multiply both
sides of the equation by A-1 and conclude that x will be the only solution.
(2) If Ax=b has a unique solution x, then we claim A cannot be singular. If A
were singular, then the equation AX=0 would have a solution z/=0, But this
would imply that y=x+z is a second solution to AX=b since Ay=A(x+z)=Ax+Az
= b+0=b
61
Corollary 1.5.3
The system of n linear equations in n unknown Ax = b has a unique solution iff A is nonsingular.
• Pf: () If A is nonsingular, and is any solution of
Ax = b, then
∴ is the unique solution.
x̂
bx ˆA
bx11 )ˆ( AAA
bx1
ˆ A
62
5. Elementary Matrices
() Suppose that Ax = b has a unique solution
and A is singular
∴ Ax = 0 has a solution z≠0 (i.e., Az = 0)
∴ Let , clear
∴ y is also the solution to Ax = b
∴ contradiction
∴ A must be nonsingular
x̂),ˆ( bx A
zxy ˆ xy ˆ
00bzxzxy AAAA ˆ)ˆ(
63
5. Elementary Matrices
If A is nonsingular, A is row equivalent to I, so there exist elementary matrices E1, E2, …, Ek such that
Ek Ek-1 … E1 A = I(Ek Ek-1 … E1 A)A
-1 = (I)A-1
Ek Ek-1 … E1 I = A-1
64
5. Elementary Matrices
The same series of elementary row operations that transform a nonsingular matrix A into I will transform I into A-1. That is, the reduced row echelon form of the augmented matrix (A | I) will be (I | A-1).
65
5. Elementary Matrices
66
5. Elementary Matrices
67
5. Elementary Matrices
If A is nonsingular, A is row equivalent to I, so there exist elementary matrices E1, E2, …, Ek such that
Ek Ek-1 … E1 A = I(Ek Ek-1 … E1 A)A
-1 = (I)A-1
Ek Ek-1 … E1 I = A-1
68
5. Elementary Matrices
The same series of elementary row operations that transform a nonsingular matrix A into I will transform I into A-1. That is, the reduced row echelon form of the augmented matrix (A | I) will be (I | A-1).
69
Example 4
Compute A-1 if
Sol:
322
021
341
A
1 100
010
001
322
021
341-AI
61
21
61
41
41
41
21
21
21
1A
70
Example 5
Solve the system:
Sol: Ax = b x = A-1b
8322
122
1234
321
21
321
xxx
xx
xxx
3
84
4
8
12
12
61
21
61
41
41
41
21
21
21
1bx A
71
Diagonal and Triangular Matrices
An n×n matrix A is said to be upper triangular if aij = 0 for i > j, eg.,
An n×n matrix A is said to be lower triangular if aij = 0 for i < j. e.g.,
500
120
123
341
006
001
72
Note
A matrix is said to be triangular if it is either upper triangular or lower triangular
73
5. Elementary Matrices
An n×n matrix A is said to be diagonal if aij = 0 whenever i ≠ j. e.g.,
Note: A diagonal matrix is both upper triangular and lower triangular.
100
030
001
74
Triangular Factorization
If an n×n matrix A can be reduced to strict upper triangular form using only row operation Ⅲ, then it is possible to represent the reduction process in terms of a matrix factorization.
75
Example 6
Ull
l
A
800
130
2423
590
130
2422
914
251
242
3231
21
21
132
01
001
1
01
001
21
3231
21
ll
lL
ALU
914
251
242
800
130
242
132
01
001
21
76
Note
L is unit lower triangular.
The factorization of the matrix A into a product of a unit lower triangular matrixL times a strictly upper triangular matrixU is often referred to as an LU factorization.
77
5. Elementary Matrices
Why LU factorization works?
Since E3 E2 E1 A = U, where
A = E1-1 E2
-1 E3-1U
E1-1 E2
-1 E3-1 =
L
130
010
001
102
010
001
100
01
001
21
,
102
010
001
2
E
130
010
001
3E,
100
01
001
21
1
E
78
6. Partitioned Matrices
A matrix is composed of a number of submatrices.
Computation Merits
Analysis Merits
79
6. Partitioned Matrices
If A is an m×n matrix and B is n×r which has been partitioned into columns [b1, b2, …, br], then the block multiplication of Atimes B is given by
AB = A [b1, b2, …, br] = [Ab1, Ab2, . . ., Abr]
80
6. Partitioned Matrices
For example
212
131A
1
5 ,
1
15 ,
2
6321 bbb AAA
1
5
1
15
2
6) , ,( hence 321 bbbA
• In particular,
[a1, a2, …, an] = A = AI = [Ae1, Ae2, …, Aen]
81
6. Partitioned Matrices
For example,
111
323 and
71
43
52
BA
494
5105
191
3
2
1
B
B
B
a
a
a
4 9 4
5105
19 1
ˆ
ˆ
ˆ
3
2
1
B
B
B
AB
a
a
a
82
6. Partitioned Matrices
Block Multiplication
83
6. Partitioned Matrices
84
6. Partitioned Matrices
85
6. Partitioned Matrices
86
6. Partitioned MatricesMultiplying Partitioned Matrices
Partitioned matrices can be multiplied as though the submatrices were scalars :
is the partitioned matrix whose ijth “entry” is the
matrix
provided that all the products AikBkj are defined.
pmpjpp
mj
mj
npnn
ipii
p
p
BBBB
BBBB
BBBB
AAA
AAA
AAA
AAA
AB
21
222221
111211
21
21
22221
11211
p
k
kjikpjipjiji BABABABA1
2211
87
Example 1
Let
Sol: (1) If
then
2123
1113
1121
1111
and
2233
1122
1111
2221
1211
BB
BBBA
2233
1122
1111
2221
1211
AA
AAA
12101518
76910
5468
2123
1113
1121
1111
2233
1122
1111
AB
88
Example 1
(2) If
then
2233
1122
1111
2221
1211
AA
AAA
12101518
76910
5468
2123
1113
1121
1111
2233
1122
1111
AB
89
Example 2
Let A be an n×n matrix of the form
where A11 is a k×k matrix (k < n). Then A is
nonsingular iff A11 and A22 are nonsingular.
Sol: () If A11 and A22 are nonsingular, then
Thus A is nonsingular.
22
11
AO
OAA
IIO
OI
AO
OA
AO
OA
I IO
OI
AO
OA
AO
OA
kn
k
-
-
kn
k
-
-
1
22
1
11
22
11
22
11
1
22
1
11 and
90
Example 2
() If A is nonsingular, let B = A-1 and
Since BA = I = AB, i.e.,
Thus,
B11A11 = Ik = A11B11B22A22 = In-k = A22B22
And hence A11 and A22 are nonsingular, with
inverse B11 and B22.
2221
1211
BB
BBB
22222122
12111111
22221121
22121111
2221
1211
22
11
22
11
2221
1211
BABA
BABA
IO
OI
ABAB
ABAB
BB
BB
AO
OA
IO
OI
AO
OA
BB
BB
n-k
k
n-k
k
91
Outer Product Expansions
Given two vectors x and y in Rn, the scalar product or the inner product is defined as the matrix product xTy, which is the product of a row vector (a 1×nmatrix) times a column vector (an n×1 matrix) and results in a 1×1 matrix or simply a scalar
nn
n
n
T yxyxyx
y
y
y
xxx , , , 22112
1
21
yx
92
Outer Product Expansions
The outer product is defined as the
matrix product xyT, which is the product
of an n×1 matrix times an 1×n matrix and
results in an n×n matrix:
Each of the rows is a multiple of yT
Each of the column vectors is a multiple of x
nnnn
n
n
n
n
T
yxyxyx
yxyxyx
yxyxyx
yyy
x
x
x
21
22212
12111
21
2
1
, , , xy
93
Outer Product Expansions
For example
2
5
3
and
3
1
4
yx
6159
253
82012
253
3
1
4T
xy
94
Outer Product Expansions
Let X be an m×n matrix and Y a k×nmatrix, the outer product expansion of Xand Y is a matrix product XYT. XYT can be calculated by partitioning X into columns and YT into rows and then perform the block multiplication:
TnnTT
T
n
T
T
n
TXY yxyxyx
y
y
y
xxx , , , 22112
1
21
95
Example 3
Given
1
4
2
3
2
1
and
2
4
1
1
2
3
YX
142
2
4
1
321
1
2
3
142
321
2
4
1
1
2
3
TXY
284
4168
142
321
642
963
96
Conclusion
Systems of Linear Equations
Row Echelon Form
Matrix Arithmetic
Matrix Algebra
Elementary Matrices
Partitioned Matrices