Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product...
Transcript of Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product...
![Page 1: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/1.jpg)
Linear algebra primer
Instructor: Taylor Berg-KirkpatrickSlides: Sanjoy Dasgupta
![Page 2: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/2.jpg)
Data as vectors and matrices
0 1 2 3 4 5 6 0
1
2
3
4
5
6
![Page 3: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/3.jpg)
Matrix-vector notation
Vector x ∈ Rd :
x =
x1x2x3...xd
Matrix M ∈ Rr×d :
M =
M11 M12 · · · M1d
M21 M22 · · · M2d...
.... . .
...Mr1 Mr2 · · · Mrd
Mij = entry at row i , column j
![Page 4: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/4.jpg)
Matrix-vector notation
Vector x ∈ Rd :
x =
x1x2x3...xd
Matrix M ∈ Rr×d :
M =
M11 M12 · · · M1d
M21 M22 · · · M2d...
.... . .
...Mr1 Mr2 · · · Mrd
Mij = entry at row i , column j
![Page 5: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/5.jpg)
Transpose of vectors and matrices
x =
1630
has transpose xT =
M =
1 2 0 43 9 1 68 7 0 2
has transpose MT = .
• (AT )ij = Aji
• (AT )T = A
![Page 6: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/6.jpg)
Adding and subtracting vectors and matrices
![Page 7: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/7.jpg)
Dot product of two vectors
Dot product of vectors x , y ∈ Rd :
x · y = x1y1 + x2y2 + · · ·+ xdyd .
What is the dot product between these two vectors?
0 1 2 3 4
1
2
3
4
-1 -2 -3 -4
x
y
![Page 8: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/8.jpg)
Dot product of two vectors
Dot product of vectors x , y ∈ Rd :
x · y = x1y1 + x2y2 + · · ·+ xdyd .
What is the dot product between these two vectors?
0 1 2 3 4
1
2
3
4
-1 -2 -3 -4
x
y
![Page 9: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/9.jpg)
Dot products and angles
Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .
Tells us the angle between x and y :
y
✓
x
cos θ =x · y‖x‖ ‖y‖
.
• x is orthogonal (at right angles) to y if and only if x · y = 0
• When x , y are unit vectors (length 1): cos θ = x · y• What is x · x?
![Page 10: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/10.jpg)
Dot products and angles
Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .
Tells us the angle between x and y :
y
✓
x
cos θ =x · y‖x‖ ‖y‖
.
• x is orthogonal (at right angles) to y if and only if x · y = 0
• When x , y are unit vectors (length 1): cos θ = x · y• What is x · x?
![Page 11: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/11.jpg)
Dot products and angles
Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .
Tells us the angle between x and y :
y
✓
x
cos θ =x · y‖x‖ ‖y‖
.
• x is orthogonal (at right angles) to y if and only if x · y = 0
• When x , y are unit vectors (length 1): cos θ = x · y
• What is x · x?
![Page 12: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/12.jpg)
Dot products and angles
Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .
Tells us the angle between x and y :
y
✓
x
cos θ =x · y‖x‖ ‖y‖
.
• x is orthogonal (at right angles) to y if and only if x · y = 0
• When x , y are unit vectors (length 1): cos θ = x · y• What is x · x?
![Page 13: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/13.jpg)
Matrix-vector product
Product of matrix M ∈ Rr×p and vector x ∈ Rp:
![Page 14: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/14.jpg)
Linear functions
Let M be any r × d matrix.The function x 7→ Mx sends Rd to Rr .
It is a linear function: M(x + x ′) = Mx + Mx ′.
E.g. M =
(2 0 −10 1 1
)
![Page 15: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/15.jpg)
Write the linear function f (x1, x2) = 3x1 + 2x2 using vectornotation.
![Page 16: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/16.jpg)
A linear function from R2 to R2 is given by M =
(1 12 2
).
As x varies, does Mx fill up all of R2?
![Page 17: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/17.jpg)
The identity matrix
The d × d identity matrix Id sends each x ∈ Rd to itself.
Id =
1 0 0 · · · 00 1 0 · · · 00 0 1 · · · 0...
......
. . ....
0 0 0 · · · 1
![Page 18: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/18.jpg)
Matrix-matrix product
Product of matrix A ∈ Rr×k and matrix B ∈ Rk×p:
![Page 19: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/19.jpg)
Matrix products
If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with
(AB)ij = (dot product of ith row of A and jth column of B)
=k∑
`=1
Ai`B`j
• IkB = B and A Ik = A
• Can check: (AB)T = BTAT
• For two vectors u, v ∈ Rd , what is uT v?
![Page 20: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/20.jpg)
Matrix products
If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with
(AB)ij = (dot product of ith row of A and jth column of B)
=k∑
`=1
Ai`B`j
• IkB = B and A Ik = A
• Can check: (AB)T = BTAT
• For two vectors u, v ∈ Rd , what is uT v?
![Page 21: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/21.jpg)
Matrix products
If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with
(AB)ij = (dot product of ith row of A and jth column of B)
=k∑
`=1
Ai`B`j
• IkB = B and A Ik = A
• Can check: (AB)T = BTAT
• For two vectors u, v ∈ Rd , what is uT v?
![Page 22: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/22.jpg)
Matrix products
If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with
(AB)ij = (dot product of ith row of A and jth column of B)
=k∑
`=1
Ai`B`j
• IkB = B and A Ik = A
• Can check: (AB)T = BTAT
• For two vectors u, v ∈ Rd , what is uT v?
![Page 23: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/23.jpg)
Some special cases
For vector x ∈ Rd , what are xT x and xxT ?
![Page 24: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/24.jpg)
Special case, generalized
For vector x ∈ Rd , we have xT x = ‖x‖2.
What about xTMx , for arbitrary d × d matrix M?
![Page 25: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/25.jpg)
Quadratic functions
Let M be any d × d (square) matrix.For x ∈ Rd , the mapping x 7→ xTMx is a quadratic functionfrom Rd to R:
xTMx =d∑
i ,j=1
Mijxixj .
What is the quadratic function associated with M =
1 0 00 2 03 4 5
?
![Page 26: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/26.jpg)
Write the quadratic function f (x1, x2) = x21 + 2x1x2 + 3x22 usingmatrices and vectors.
![Page 27: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/27.jpg)
Special cases of square matrices
• Symmetric: M = MT1 2 32 4 53 5 6
,
1 2 31 2 43 4 6
• Diagonal: M = diag(m1,m2, . . . ,md)
diag(1, 4, 7) =
1 0 00 4 00 0 7
![Page 28: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/28.jpg)
Determinant of a square matrix
Determinant of A =
(a bc d
)is |A| = ad − bc.
Example: A =
(3 11 2
)
![Page 29: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/29.jpg)
Inverse of a square matrix
The inverse of a d × d matrix A is a d × d matrix B for whichAB = BA = Id .Notation: A−1.
Example: if A =
(1 2−2 0
)then A−1 =
(0 −1/2
1/2 1/4
). Check!
![Page 30: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/30.jpg)
Inverse of a square matrix
The inverse of a d × d matrix A is a d × d matrix B for whichAB = BA = Id .Notation: A−1.
Example: if A =
(1 2−2 0
)then A−1 =
(0 −1/2
1/2 1/4
). Check!
![Page 31: Instructor: Taylor Berg-Kirkpatrick Slides: Sanjoy Dasgupta · Dot products and angles Dot product of vectors x;y 2Rd: x y = x 1y 1 + x 2y 2 + + x dy d: Tells us the angle between](https://reader034.fdocuments.us/reader034/viewer/2022042106/5e859ea14f8b653695762602/html5/thumbnails/31.jpg)
Inverse of a square matrix, cont’d
The inverse of a d × d matrix A is a d × d matrix B for whichAB = BA = Id .Notation: A−1.
• Not all square matrices have an inverse
• Square matrix A is invertible if and only if |A| 6= 0
• What is the inverse of A = diag(a1, . . . , ad)?