Linear Algebra 1 -...
Transcript of Linear Algebra 1 -...
![Page 1: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/1.jpg)
Linear Algebra 1
Industrial AI Lab.
Prof. Seungchul Lee
Some materials from linear algebra review by Prof. Zico Kolter from CMU
![Page 2: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/2.jpg)
Linear Equations
• Set of linear equations (two equations, two unknowns)
2
1 2
1 2
4 5 13
2 3 9
x x
x x
![Page 3: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/3.jpg)
Linear Equations
• Solving linear equations– Two linear equations
– In a vector form, 𝐴𝑥 = 𝑏, with
3
1 2
1 2
4 5 13
2 3 9
x x
x x
1
2
4 5 13, ,
2 3 9
xA x b
x
![Page 4: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/4.jpg)
Linear Equations
• Solving linear equations– Two linear equations
– In a vector form, 𝐴𝑥 = 𝑏, with
– Solution using inverse
4
1 2
1 2
4 5 13
2 3 9
x x
x x
1
2
4 5 13, ,
2 3 9
xA x b
x
1 1
1
Ax b
A Ax A b
x A b
![Page 5: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/5.jpg)
Linear Equations
• Solving linear equations– Two linear equations
– In a vector form, 𝐴𝑥 = 𝑏, with
– Solution using inverse
– Don’t worry here about how to compute matrix inverse
– We will use a numpy to compute
5
1 2
1 2
4 5 13
2 3 9
x x
x x
1
2
4 5 13, ,
2 3 9
xA x b
x
1 1
1
Ax b
A Ax A b
x A b
![Page 6: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/6.jpg)
Linear Equations in Python
6
1 2
1 2
4 5 13
2 3 9
x x
x x
1
2
4 5 13, ,
2 3 9
xA x b
x
![Page 7: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/7.jpg)
System of Linear Equations
• Consider a system of linear equations
• Can be written in a matrix form as 𝑦 = 𝐴𝑥, where
7
![Page 8: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/8.jpg)
Elements of a Matrix
• Can write a matrix in terms of its columns
• Careful, 𝑎𝑖 here corresponds to an entire vector 𝑎𝑖 ∈ ℝ𝑚
• Similarly, can write a matrix in terms of rows
• 𝑏𝑖 ∈ ℝ𝑛
8
![Page 9: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/9.jpg)
Vector-Vector Products
• Inner product: 𝑥, 𝑦 ∈ ℝ𝑛
9
![Page 10: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/10.jpg)
Matrix-Vector Products
• 𝐴 ∈ ℝ𝑚×𝑛, 𝑥 ∈ ℝ𝑛 ⟺ 𝐴𝑥 ∈ ℝ𝑚
• Writing 𝐴 by rows, each entry of 𝐴𝑥 is an inner product between 𝑥 and a row of 𝐴
10
![Page 11: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/11.jpg)
Matrix-Vector Products
• 𝐴 ∈ ℝ𝑚×𝑛, 𝑥 ∈ ℝ𝑛 ⟺ 𝐴𝑥 ∈ ℝ𝑚
• Writing 𝐴 by columns, 𝐴𝑥 is a linear combination of the columns of 𝐴, with coefficients given by 𝑥
11
![Page 12: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/12.jpg)
Symmetric Matrices
• Symmetric matrix:
𝐴 ∈ ℝ𝑛×𝑛 with 𝐴 = 𝐴𝑇
• Arise naturally in many settings
• For 𝐴 ∈ ℝ𝑚×𝑛,
𝐴𝑇𝐴 ∈ ℝ𝑛×𝑛 is symmetric
12
![Page 13: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/13.jpg)
Norms (Strength or Distance in Linear Space)
• A vector norm is any function 𝑓:ℝ𝑛 ⟹ℝ with
• 𝑙2 norm
• 𝑙1 norm
• 𝑥 measures length of vector (from origin)
13
![Page 14: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/14.jpg)
Norms in Python
14
![Page 15: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/15.jpg)
Orthogonality
• Two vectors 𝑥, 𝑦 ∈ ℝ𝑛 are orthogonal if
• They are orthonormal if
15
and
![Page 16: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/16.jpg)
Angle between Vectors
• For any 𝑥, 𝑦 ∈ ℝ𝑛,
• (unsigned) angle between vectors in ℝ𝑛 defined as
16
![Page 17: Linear Algebra 1 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/02_Linear... · Linear Algebra 1 Industrial AI Lab. Prof. Seungchul Lee Some materials](https://reader030.fdocuments.us/reader030/viewer/2022040716/5e20caff8d7783242f67b85f/html5/thumbnails/17.jpg)
Angle between Vectors
• ȁ𝑥 𝑥𝑇𝑦 ≤ 0 defines a half space with outward normal vector 𝑦, and boundary passing through 0
17