Lecture -- Review of Linear Algebra · Review of Basic Linear Algebra Outline •Solving systems of...
19
10/2/2019 1 Computational Science: Computational Methods in Engineering Review of Basic Linear Algebra Outline • Solving systems of linear equations • Matrix terminology and special matrices • Matrix operations • Common linear algebra problems 2 1 2
Transcript of Lecture -- Review of Linear Algebra · Review of Basic Linear Algebra Outline •Solving systems of...
10/2/2019
1
Computational Science:
Computational Methods in Engineering
Review of Basic Linear Algebra
Outline
• Solving systems of linear equations
•Matrix terminology and special matrices
•Matrix operations
•Common linear algebra problems
2
1
2
10/2/2019
2
3
Solving Systems of Linear Equations
Systems of Linear Equations
4
Very often in science and engineering, problems can be reduced to a system of linear equations.
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
n n nn n n
a x a x a x b
a x a x a x b
a x a x a x b
constant coefficient (usually known)
unknown values
constants (usually excitation)
ij
i
i
a
x
b
3
4
10/2/2019
3
Direct Analytical Solution
5
Step 1 – Solve first equation for x1.
Suppose it is desired to solve the following system of equations
11 1 12 2 13 3 1
21 1 22 2 23 3 2
31 1 32 2 33 3 3
a x a x a x b
a x a x a x b
a x a x a x b
131 1211 1 12 2 13 3 1 1 2 3
11 11 11
21 1 22 2 23 3 2
31 1 32 2 33 3 3
ab a
a x a x a x b x x xa a a
a x a x a x b
a x a x a x b
Direct Analytical Solution
6
Step 2 – Forward Substitution – Substitute this new equation into 2nd and 3rd equations to eliminate x1.
11 1 12 2 13 3 1
22 2 23 3 2
32 2 33 3 3
a x a x a x b
a x a x b
a x a x b
21 1321 12 21 122 22 23 23 2 2
11 11 11
31 12 31 13 31 132 32 33 33 3 3
11 11 11
a aa a a ba a a a b b
a a a
a a a a a ba a a a b b
a a a
5
6
10/2/2019
4
Direct Analytical Solution
7
Step 3 – Solve second equation for x2.
11 1 12 2 13 3 1
23222 2 23 3 2 2 3
22 22
32 2 33 3 3
a x a x a x b
aba x a x b x x
a a
a x a x b
Direct Analytical Solution
8
Step 4 – Forward Substitution – Substitute this new equation into 3rd equation to eliminate x2.
11 1 12 2 13 3 1
22 2 23 3 2
33 3 3
a x a x a x b
a x a x b
a x b
32 23 32 233 33 3 3
22 22
a a a b
a a b ba a
7
8
10/2/2019
5
Direct Analytical Solution
9
Step 5 – Solve third equation for x3. Since this is the last equation, the final answer for is obtained for x3.
33
33
bx
a
Direct Analytical Solution
10
Step 6 – Backward Substitution – Given x3, calculate x2 using equation from Step 3.
2 23 32
22
b a xx
a
9
10
10/2/2019
6
Direct Analytical Solution
11
Step 7 – Backward Substitution – Given x2 and x3, calculate x1 using equation from Step 1.
1 12 2 13 31
11
b a x a xx
a
12
Matrix Terminology& Special Matrices
11
12
10/2/2019
7
Systems of Linear Equations
13
Systems of equations can be written in matrix form.
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
n n nn n n
a x a x a x b
a x a x a x b
a x a x a x b
11 12 1 1 1
21 22 2 2 2
1 2
n
n
n n nn n n
a a a x b
a a a x b
a a a x b
or
A
A
or
x
x
or
b
b
Rows, Columns, and Diagonals
14
The center diagonal is usually just called the diagonal.
The elements along the diagonal are sometimes called the pivot elements.
13
14
10/2/2019
8
Special Matrices (1 of 2)
15
Symmetric Matrix
1 2 9 4
2 6 5 8
9 5 7 0
4 8 0 3
A
Diagonal Matrix
1 0 0 0
0 6 0 0
0 0 7 0
0 0 0 3
A
Identity Matrix
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
I
Zero Matrix
0 0 0 0
0 0 0 00
0 0 0 0
0 0 0 0
Bandwidth of 3
Special Matrices (2 of 2)
16
Upper Triangular Matrix
1 2 9 4
0 6 5 8
0 0 7 5
0 0 0 3
A
Lower Triangular Matrix
1 0 0 0
2 6 0 0
9 5 7 0
4 8 1 3
A
Banded Matrix
1 2 0 0
4 6 5 0
0 8 7 5
0 0 10 3
A
Vandermonde Matrix2
1 1 12
2 2 22
3 3 3
21 1 1
1
1
1
1
N
N
N
NN N N
x x x
x x x
x x x
x x x
Arises when curve fitting to polynomials.Usually ill‐conditioned for large matrices.
Triangular matrices can be thought of as “almost” solved matrices. They are very fast to solve.
15
16
10/2/2019
9
Block Matrices
17
Block matrices are “matrices of matrices.”
11 12 11 12 11 12 11 12
21 22 21 22 21 22 21 22
a a b b c c d d
A B C Da a b b c c d d
11 12 11 12
21 22 21 22
11 12 11 12
21 22 21 22
a a b b
a a b bA BF
C D c c d d
c c d d
Sparse Matrices
18
Many matrices contain 99.9% zeros.
It is not efficient use of memory to store all these zeros. Instead, we store only the non‐zero elements along with their positions in the matrix.
The opposite of a sparse matrix is a full matrix.
17
18
10/2/2019
10
Matrix Problem Size
19
# Equations > # Unknowns # Equations = # Unknowns # Equations < # Unknowns
Usually occurs when the equations are derived from samples.
Solution is obtained as a best fit and is not exact.
Applications• Curve fitting
Most usual case.
Many standard algorithms exist to obtain an exact solution.
Applications• Circuit theory• Solving ODEs
Usually occurs when little is known about the problem or solution.
Solution is obtained by optimization and is not exact.
Applications• Topology optimization
Health of a Matrix (1 of 3)
20
Is this system of equations solvable?
2 8 1 2 1 8
2 8 1 2 1 8
3 4 3 1 1 4
x y z x
x y z y
x y z z
No!The 1st and 2nd equations are the same. The 2nd
equation does not provide any new information to the problem.
2 8 1 2 1 8
2 4 2 16 2 4 2 16
3 4 3 1 1 4
x y z x
x y z y
x y z z
2 8 1 2 1 8
4 2 12 4 1 2 12
3 4 3 1 1 4
x y z x
x y z y
x y z z
No!The 2nd equation is just 2× the 1st equation. The 2nd
equation is still not providing any new information.
No!The 2nd equation is the sum of the 1st and 3rd equation, thus the 2nd equation still does not provide any new information.
19
20
10/2/2019
11
Health of a Matrix (2 of 3)
21
Is this system of equations solvable?
8 1 0 1 8
2 7 1 0 2 7
3 4 3 0 1 4
x z x
x z y
x z z
No!None of these equations contain any information about y.
So how do we know if a problem is solvable?
• All rows must be linearly independent – this ensures they provide new information to the problem.• No rows can be all zeros – This would not provide any information.• No columns can be all zeros – This would be ignoring information from one of the unknowns.
is solvable if det 0A x b A
Health of a Matrix (3 of 3)
22
Is the following system of equations solvable?
Condition Number of a Matrix
max
min
AA
A
2 8 1 2 1 8
1.0001 2 8.0001 1.0001 2 1 8.0001
3 4 3 1 1 4
x y z x
x y z y
x y z z
Technically yes, but we would expect the solution to be somewhat “touchy” and unstable. This is an ill-conditionedmatrix.
The condition number (A) of matrix A is a measure of how much an answer will change given small changes in the matrix b.
Matrices with high condition numbers are less stable. Small changes in the element values of A will result in large changes in the elements of b.
min
max
smallest singular value of
largest singular value of
A A
A A
21
22
10/2/2019
12
Example: Condition Number
23
What is the condition number?
1 2 1
1 2 1
3 1 1
A 165.84 10 A
1 2 1
1 0 1
3 1 1
A 7.76 A
1 2 1
1.0001 2 1
3 1 1
A 51.4 10 A
1 2 1
1.01 2 1
3 1 1
A 31.4 10 A
24
Matrix Operations
23
24
10/2/2019
13
Matrix Math (1 of 4)
25
Addition:
11 12 13 11 12 13 11 11 12 12 13 13
21 22 23 21 22 23 21 21 22 22 23 23
31 32 33 31 32 33 31 31 32 32 33 33
a a a b b b a b a b a b
A B a a a b b b a b a b a b
a a a b b b a b a b a b
11 12 13 11 12 13 11 11 12 12 13 13
21 22 23 21 22 23 21 21 22 22 23 23
31 32 33 31 32 33 31 31 32 32 33 33
a a a b b b a b a b a b
A B a a a b b b a b a b a b
a a a b b b a b a b a b
Subtraction:
Matrix Math (2 of 4)
26
Multiplication by a Scalar:
11 12 13 11 12 13
21 22 23 21 22 23
31 32 33 31 32 33
a a a sa sa sa
s A s a a a sa sa sa
a a a sa sa sa
11 12 13 11 12 13 11 11 12 21 13 31
21 22 23 21 22 23
31 32 33 31 32 33
11 12 13 1 11 1 12 2 13 3
21 22 23 2 21 1 22
31 32 33 3
# #
# # #
# # #
a a a b b b a b a b a b
A B a a a b b b
a a a b b b
a a a x a x a x a x
A x a a a x a x a x
a a a x
2 23 3
31 1 32 2 33 3
a x
a x a x a x
Multiplication by a Matrix
25
26
10/2/2019
14
Matrix Math (3 of 4)
27
Matrix Transpose:
11 12 13 11
21 22 23 22
31 32 3
21 31
12 32
13 2 333 3
T
T
a a a a
A a a a a
a a a a
a a
a a
a a
* *21 31
* *1
**
11 12 13 11* *
21 22 23 222 32* *1
*31 32 33 333 23
T
H
ij ji
a a
a a
a a
a a a a
A a a a a a a
a a a a
Hermitian Transpose:
ij jia a
Animation of Transpose Operation
Matrix Math (4 of 4)
28
Determinants:
det A Think of this as the “magnitude” or “volume” of a matrix.
Matrix Inverse:
1A A I
Matrix Division:
1
1
predivide
postdivide
A B
B A
Matrix Multiplication:
premultiplies
postmultiplies
A B A B
B A A B
A\B
B/AWhile both expressions divide by [A], these do not give the same answer.
27
28
10/2/2019
15
Matrix Algebra (1 of 3)
29
Commutative Laws
A B B A
A B B A
Associative Laws
A B C A B C
A B C A B C
Distributive Laws
A B C A C B C
A B C A B A C
Matrix Inverses and Transposes
1 1
11
1 1 1
1 1
TT
T T T
A A A A I
A A
A B B A
A A
A B A B
TT
T T T
A A
A B B A
[A][B] = [B][A] only when [A]and [B] are diagonal matrices.
Matrix Algebra (2 of 3)
30
Addition with a Scalar
11 12 1
21 22 2
1 2
doesn't make sense
n
n
n n nn
A
a a a
a a aI A
a a a
Multiplication with a Scalar
A B A B
A B A B A B
29
30
10/2/2019
16
Matrix Algebra (3 of 3)
31
Operations with Special Matrices
0 0 0
0 0
0
A A
I A A I A
A A A
A A
Example of Matrix Algebra
32
Simplify the Following Equation
11 C A D BC D
Step 1 – Subtract D from both sides
11
11
11
C A D D BC D D
C A 0 BC 0
C A BC
Step 2 – Inverse both sides
1 111
1 1 1
C A
C A C B
BC
Step 3 – Premultiply both sides by C.1 1 1
1
1
CC A CC B
IA IB
A B
31
32
10/2/2019
17
33
Common Linear Algebra Problems
[A][x] = [b]
34
This problem arises when a problem [A] is given some excitation [b] and produces a solution [x].
Examples: (1) waves scattering from an object, (2) heat through a device, (3) solving currents and voltages in a circuit.
It produces a single solution.
Step 1 – Differential equation2
2
d f dff b
dx dx
Step 2 – ODE is converted to system of equations using finite‐differences, finite elements, etc.
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
n n
n n
n n nn n n
a f a f a f b
a f a f a f b
a f a f a f b
Step 3 – System of equations is put into matrix form.
11 12 1 1 1
21 22 2 2 2
1 2
n
n
n n nn n n
a a a f b
a a a f b
a a a f b
Step 4 – Matrix problem is solved for [f]
1f A b
Step 5 – [f] is post processed to learn something.
33
34
10/2/2019
18
Eigen‐Value Problems 𝐴 𝑥 𝜆 𝑥
35
Eigen‐value problems arise when multiple solutions exist. No excitation is needed.
Examples: (1) resonating modes on a string, (2) electromagnetic modes in a waveguide, (3) electronic bands in a semiconductor.
Standard eigen-value problem
Generalized eigen-value problem
A x x
A x B x
is the linear operation
is the unknown (eigen-vector)
is the eigen-value and is just a scalar number
is potentially another part of the linear operation
A
x
B
Determinants
36
The determinant is an important number associated with square matrices.
It is sort of a magnitude or volume.
Unique solutions to systems of equations do not exist when the determinant is zero.
11 1211 22 12 21
21 22
deta a
A a a a aa a
3×3 Matrices
2×2 Matrices
11 12 13
22 23 21 23 21 2221 22 23 11 12 13
32 33 31 33 31 3231 32 33
det
a a aa a a a a a
A a a a a a aa a a a a a
a a a
This can be calculated by walking across any of the rows.
35
36
10/2/2019
19
Cramer’s Rule
37
Cramer’s rule provides a methodical approach for calculating the unknowns of a system of equations.
11 12 13 1 1
21 22 23 2 2
31 32 33 3 3
a a a x b
a a a x b
a a a x b
1 12 13
1 2 22 23
3 32 33
1b a a
x b a aD
b a a
11 1 13
2 21 2 23
31 3 33
1a b a
x a b aD
a b a
11 12 1
3 21 22 2
31 32 3
1a a b
x a a bD
a a b
11 12 13 11 12 13
21 22 23 21 22 23
31 32 33 31 32 33
det
a a a a a a
D a a a a a a
a a a a a a
37
