Linear Algebra 3 -...
Transcript of Linear Algebra 3 -...
![Page 1: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/1.jpg)
Linear Algebra 3
Industrial AI Lab.
Prof. Seungchul Lee
![Page 2: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/2.jpg)
System of Linear Equations
• Well-determined linear systems
• Under-determined linear systems
• Over-determined linear systems
2
![Page 3: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/3.jpg)
Well-Determined Linear Systems
• System of linear equations
• Geometric interpretation
3
![Page 4: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/4.jpg)
Well-Determined Linear Systems
• System of linear equations
• Generalization
4
![Page 5: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/5.jpg)
Under-Determined Linear Systems
• System of linear equations
• Geometric interpretation
5
![Page 6: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/6.jpg)
Under-Determined Linear Systems
• System of linear equations
• Generalization
6
![Page 7: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/7.jpg)
Over-Determined Linear Systems
• System of linear equations
• Geometric interpretation
7
![Page 8: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/8.jpg)
Over-Determined Linear Systems
• System of linear equations
• Generalization
8
![Page 9: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/9.jpg)
Summary of Linear Systems
• Square: Well-determined
• Fat: Under-determined
• Skinny: Over-determined
9
![Page 10: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/10.jpg)
Least-Norm Solution
• For Under-Determined Linear System
• Find the solution of 𝐴𝑋 = 𝐵 that minimize 𝑋 or 𝑋 2
• i.e., optimization problem
10
![Page 11: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/11.jpg)
Least-Norm Solution
• Optimization problem
• Geometric interpretation
• Select one solution among many solutions
• Often control problem
11
![Page 12: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/12.jpg)
Least-Square Solution
• For Over-Determined Linear System
• Find 𝑋 that minimizes 𝐸 or 𝐸 2 (error)
• i.e. optimization problem
12
![Page 13: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/13.jpg)
Least-Square Solution
• i.e. optimization problem
• Geometric interpretation
• Often estimation problem
13
![Page 14: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/14.jpg)
Vector Projection onto Y
• The vector projection of a vector 𝑋 on (or onto) a nonzero vector 𝑌 is the orthogonal projection of 𝑋onto a straight line parallel to 𝑌
14
![Page 15: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/15.jpg)
Vector Projection onto Y
• Another way of computing 𝜔 and 𝑊
15
![Page 16: Linear Algebra 3 - i-systems.github.ioi-systems.github.io/HSE545/iAI/ML/topics/02_Linear_Algebra/04... · System of Linear Equations •Well-determined linear systems •Under-determined](https://reader031.fdocuments.us/reader031/viewer/2022022117/5ca5ac7a88c9938b538cb603/html5/thumbnails/16.jpg)
Orthogonal Projection onto a Subspace
• Projection of 𝐵 onto a subspace 𝑈 of span of 𝐴1 and 𝐴2• Orthogonality
16