Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I...
Transcript of Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I...
![Page 1: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/1.jpg)
Vector spaces
DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysishttp://www.cims.nyu.edu/~cfgranda/pages/OBDA_fall17/index.html
Carlos Fernandez-Granda
![Page 2: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/2.jpg)
![Page 3: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/3.jpg)
Vector space
Consists of:
I A set V
I A scalar field (usually R or C)
I Two operations + and ·
![Page 4: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/4.jpg)
Properties
I For any ~x , ~y ∈ V, ~x + ~y belongs to V
I For any ~x ∈ V and any scalar α, α · ~x ∈ V
I There exists a zero vector ~0 such that ~x +~0 = ~x for any ~x ∈ V
I For any ~x ∈ V there exists an additive inverse ~y such that ~x + ~y = ~0,usually denoted by −~x
![Page 5: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/5.jpg)
Properties
I The vector sum is commutative and associative, i.e. for all ~x , ~y , ~z ∈ V
~x + ~y = ~y + ~x , (~x + ~y) + ~z = ~x + (~y + ~z)
I Scalar multiplication is associative, for any scalars α and β and any~x ∈ V
α (β · ~x) = (αβ) · ~x
I Scalar and vector sums are both distributive, i.e. for any scalars α andβ and any ~x , ~y ∈ V
(α+ β) · ~x = α · ~x + β · ~x , α · (~x + ~y) = α · ~x + α · ~y
![Page 6: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/6.jpg)
Subspaces
A subspace of a vector space V is any subset of V that is also itselfa vector space
![Page 7: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/7.jpg)
Linear dependence/independence
A set of m vectors ~x1, ~x2, . . . , ~xm is linearly dependent if there existm scalar coefficients α1, α2, . . . , αm which are not all equal to zero and
m∑i=1
αi ~xi = ~0
Equivalently, any vector in a linearly dependent set can beexpressed as a linear combination of the rest
![Page 8: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/8.jpg)
Span
The span of {~x1, . . . , ~xm} is the set of all possible linear combinations
span (~x1, . . . , ~xm) :=
{~y | ~y =
m∑i=1
αi ~xi for some scalars α1, α2, . . . , αm
}
The span of any set of vectors in V is a subspace of V
![Page 9: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/9.jpg)
Basis and dimension
A basis of a vector space V is a set of independent vectors {~x1, . . . , ~xm}such that
V = span (~x1, . . . , ~xm)
If V has a basis with finite cardinality then every basis containsthe same number of vectors
The dimension dim (V) of V is the cardinality of any of its bases
Equivalently, the dimension is the number of linearly independent vectorsthat span V
![Page 10: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/10.jpg)
Standard basis
~e1 =
10...0
, ~e2 =
01...0
, . . . , ~en =
00...1
The dimension of Rn is n
![Page 11: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/11.jpg)
![Page 12: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/12.jpg)
Inner product
Operation 〈·, ·〉 that maps a pair of vectors to a scalar
![Page 13: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/13.jpg)
Properties
I If the scalar field is R, it is symmetric. For any ~x , ~y ∈ V
〈~x , ~y〉 = 〈~y , ~x〉
If the scalar field is C, then for any ~x , ~y ∈ V
〈~x , ~y〉 = 〈~y , ~x〉,
where for any α ∈ C α is the complex conjugate of α
![Page 14: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/14.jpg)
Properties
I It is linear in the first argument, i.e. for any α ∈ R and any ~x , ~y , ~z ∈ V
〈α~x , ~y〉 = α 〈~x , ~y〉 ,〈~x + ~y , ~z〉 = 〈~x , ~z〉+ 〈~y , ~z〉 .
If the scalar field is R, it is also linear in the second argument
I It is positive definite: 〈~x , ~x〉 is nonnegative for all ~x ∈ V and if〈~x , ~x〉 = 0 then ~x = ~0
![Page 15: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/15.jpg)
Dot product
Inner product between ~x , ~y ∈ Rn
~x · ~y :=∑i
~x [i ] ~y [i ]
Rn endowed with the dot product is usually called a Euclidean space ofdimension n
If ~x , ~y ∈ Cn
~x · ~y :=∑i
~x [i ] ~y [i ]
![Page 16: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/16.jpg)
Sample covariance
Quantifies joint fluctuations of two quantities or features
For a data set (x1, y1), (x2, y2), . . . , (xn, yn)
cov ((x1, y1) , . . . , (xn, yn)) :=1
n − 1
n∑i=1
(xi − av (x1, . . . , xn)) (yi − av (y1, . . . , yn))
where the average or sample mean is defined by
av (a1, . . . , an) :=1n
n∑i=1
ai
If (x1, y1), (x2, y2), . . . , (xn, yn) are iid samples from x and y
E (cov ((x1, y1) , . . . , (xn, yn))) = Cov (x, y) := E ((x− E (x)) (y − E (y)))
![Page 17: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/17.jpg)
Matrix inner product
The inner product between two m × n matrices A and B is
〈A,B〉 := tr(ATB
)=
m∑i=1
n∑j=1
AijBij
where the trace of an n × n matrix is defined as the sum of its diagonal
tr (M) :=n∑
i=1
Mii
For any pair of m × n matrices A and B
tr(BTA
):= tr
(ABT
)
![Page 18: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/18.jpg)
Function inner product
The inner product between two complex-valued square-integrablefunctions f , g defined in an interval [a, b] of the real line is
~f · ~g :=
∫ b
af (x) g (x) dx
![Page 19: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/19.jpg)
![Page 20: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/20.jpg)
Norm
Let V be a vector space, a norm is a function ||·|| from V to R withthe following properties
I It is homogeneous. For any scalar α and any ~x ∈ V
||α~x || = |α| ||~x || .
I It satisfies the triangle inequality
||~x + ~y || ≤ ||~x ||+ ||~y || .
In particular, ||~x || ≥ 0
I ||~x || = 0 implies ~x = ~0
![Page 21: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/21.jpg)
Inner-product norm
Square root of inner product of vector with itself
||~x ||〈·,·〉 :=√〈~x , ~x〉
![Page 22: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/22.jpg)
Inner-product norm
I Vectors in Rn or Cn: `2 norm
||~x ||2 :=√~x · ~x =
√√√√ n∑i=1
~x [i ]2
I Matrices in Rm×n or Cm×n: Frobenius norm
||A||F :=√
tr (ATA) =
√√√√ m∑i=1
n∑j=1
A2ij
I Square-integrable complex-valued functions: L2 norm
||f ||L2:=√〈f , f 〉 =
√∫ b
a|f (x)|2 dx
![Page 23: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/23.jpg)
Cauchy-Schwarz inequality
For any two vectors ~x and ~y in an inner-product space
|〈~x , ~y〉| ≤ ||~x ||〈·,·〉 ||~y ||〈·,·〉
Assume ||~x ||〈·,·〉 6= 0, then
〈~x , ~y〉 = − ||~x ||〈·,·〉 ||~y ||〈·,·〉 ⇐⇒ ~y = −||~y ||〈·,·〉||~x ||〈·,·〉
~x
〈~x , ~y〉 = ||~x ||〈·,·〉 ||~y ||〈·,·〉 ⇐⇒ ~y =||~y ||〈·,·〉||~x ||〈·,·〉
~x
![Page 24: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/24.jpg)
Sample variance and standard deviation
The sample variance quantifies fluctuations around the average
var (x1, x2, . . . , xn) :=1
n − 1
n∑i=1
(xi − av (x1, x2, . . . , xn))2
If x1, x2, . . . , xn are iid samples from x
E (var (x1, x2, . . . , xn)) = Var (x) := E((x− E (x))2
)The sample standard deviation is
std (x1, x2, . . . , xn) :=√
var (x1, x2, . . . , xn)
![Page 25: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/25.jpg)
Correlation coefficient
Normalized covariance
ρ(x1,y1),...,(xn,yn) :=cov ((x1, y1) , . . . , (xn, yn))
std (x1, . . . , xn) std (y1, . . . , yn)
Corollary of Cauchy-Schwarz
−1 ≤ ρ(x1,y1),...,(xn,yn) ≤ 1
and
ρ~x ,~y = −1 ⇐⇒ yi = av (y1, . . . , yn)−std (y1, . . . , yn)
std (x1, . . . , xn)(xi − av (x1, . . . , xn))
ρ~x ,~y = 1 ⇐⇒ yi = av (y1, . . . , yn) +std (y1, . . . , yn)
std (x1, . . . , xn)(xi − av (x1, . . . , xn))
![Page 26: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/26.jpg)
Correlation coefficient
ρ~x ,~y 0.50 0.90 0.99
ρ~x ,~y 0.00 -0.90 -0.99
![Page 27: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/27.jpg)
Temperature data
Temperature in Oxford over 150 yearsI Feature 1: Temperature in JanuaryI Feature 1: Temperature in August
ρ = 0.269
16 18 20 22 24 26 28
August
8
10
12
14
16
18
20
Apri
l
![Page 28: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/28.jpg)
Temperature data
Temperature in Oxford over 150 years (monthly)I Feature 1: Maximum temperatureI Feature 1: Minimum temperature
ρ = 0.962
5 0 5 10 15 20 25 30
Maximum temperature
10
5
0
5
10
15
20
Min
imum
tem
pera
ture
![Page 29: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/29.jpg)
Parallelogram law
A norm ‖ · ‖ on a vector space V is an inner-product norm if and only if
2‖~x‖2 + 2‖~y‖2 = ‖~x − ~y‖2 + ‖~x + ~y‖2
for any ~x , ~y ∈ V
![Page 30: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/30.jpg)
`1 and `∞ norms
Norms in Rn or Cn not induced by an inner product
||~x ||1 :=n∑
i=1
|~x [i ]|
||~x ||∞ := maxi|~x [i ]|
Hölder’s inequality
|〈~x , ~y〉| ≤ ||~x ||1 ||~y ||∞
![Page 31: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/31.jpg)
Norm balls
`1 `2 `∞
![Page 32: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/32.jpg)
Distance
The distance between two vectors ~x and ~y induced by a norm ||·|| is
d (~x , ~y) := ||~x − ~y ||
![Page 33: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/33.jpg)
Classification
Aim: Assign a signal to one of k predefined classes
Training data: n pairs of signals (represented as vectors) andlabels: {~x1, l1}, . . . , {~xn, ln}
![Page 34: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/34.jpg)
Nearest-neighbor classification
nearest neighbor
![Page 35: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/35.jpg)
Face recognition
Training set: 360 64× 64 images from 40 different subjects (9 each)
Test set: 1 new image from each subject
We model each image as a vector in R4096 and use the `2-norm distance
![Page 36: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/36.jpg)
Face recognition
Training set
![Page 37: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/37.jpg)
Nearest-neighbor classification
Errors: 4 / 40
Testimage
Closestimage
![Page 38: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/38.jpg)
![Page 39: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/39.jpg)
Orthogonality
Two vectors ~x and ~y are orthogonal if and only if
〈~x , ~y〉 = 0
A vector ~x is orthogonal to a set S, if
〈~x , ~s〉 = 0, for all ~s ∈ S
Two sets of S1,S2 are orthogonal if for any ~x ∈ S1, ~y ∈ S2
〈~x , ~y〉 = 0
The orthogonal complement of a subspace S is
S⊥ := {~x | 〈~x , ~y〉 = 0 for all ~y ∈ S}
![Page 40: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/40.jpg)
Pythagorean theorem
If ~x and ~y are orthogonal
||~x + ~y ||2〈·,·〉 = ||~x ||2〈·,·〉 + ||~y ||2〈·,·〉
![Page 41: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/41.jpg)
Orthonormal basis
Basis of mutually orthogonal vectors with inner-product normequal to one
If {~u1, . . . , ~un} is an orthonormal basis of a vector space V,for any ~x ∈ V
~x =n∑
i=1
〈~ui , ~x〉 ~ui
![Page 42: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/42.jpg)
Gram-Schmidt
Builds orthonormal basis from a set of linearly independent vectors~x1, . . . , ~xm in Rn
1. Set ~u1 := ~x1/ ||~x1||2
2. For i = 1, . . . ,m, compute
~vi := ~xi −i−1∑j=1
〈~uj , ~xi 〉 ~uj
and set ~ui := ~vi/ ||~vi ||2
![Page 43: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/43.jpg)
![Page 44: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/44.jpg)
Direct sum
For any subspaces S1,S2 such that
S1 ∩ S2 = {0}
the direct sum is defined as
S1 ⊕ S2 := {~x | ~x = ~s1 + ~s2 ~s1 ∈ S1, ~s2 ∈ S2}
Any vector ~x ∈ S1 ⊕ S2 has a unique representation
~x = ~s1 + ~s2 ~s1 ∈ S1, ~s2 ∈ S2
![Page 45: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/45.jpg)
Orthogonal projection
The orthogonal projection of ~x onto a subspace S is a vectordenoted by PS ~x such that
~x − PS ~x ∈ S⊥
The orthogonal projection is unique
![Page 46: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/46.jpg)
Orthogonal projection
![Page 47: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/47.jpg)
Orthogonal projection
Any vector ~x can be decomposed into
~x = PS ~x + PS⊥ ~x .
For any orthonormal basis ~b1, . . . , ~bm of S,
PS ~x =m∑i=1
⟨~x , ~bi
⟩~bi
The orthogonal projection is a linear operation. For ~x and ~y
PS (~x + ~y) = PS ~x + PS ~y
![Page 48: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/48.jpg)
Dimension of orthogonal complement
Let V be a finite-dimensional vector space, for any subspace S ⊆ V
dim (S) + dim(S⊥)= dim (V)
![Page 49: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/49.jpg)
Orthogonal projection is closest
The orthogonal projection PS ~x of a vector ~x onto a subspace S isthe solution to the optimization problem
minimize~u
||~x − ~u||〈·,·〉subject to ~u ∈ S
![Page 50: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/50.jpg)
Proof
Take any point ~s ∈ S such that ~s 6= PS ~x
||~x − ~s||2〈·,·〉
= ||~x − PS ~x + PS ~x − ~s||2〈·,·〉= ||~x − PS ~x ||2〈·,·〉 + ||PS ~x − ~s||2〈·,·〉> ||~x − PS ~x ||2〈·,·〉 if ~s 6= PS ~x
![Page 51: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/51.jpg)
Proof
Take any point ~s ∈ S such that ~s 6= PS ~x
||~x − ~s||2〈·,·〉 = ||~x − PS ~x + PS ~x − ~s||2〈·,·〉
= ||~x − PS ~x ||2〈·,·〉 + ||PS ~x − ~s||2〈·,·〉> ||~x − PS ~x ||2〈·,·〉 if ~s 6= PS ~x
![Page 52: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/52.jpg)
Proof
Take any point ~s ∈ S such that ~s 6= PS ~x
||~x − ~s||2〈·,·〉 = ||~x − PS ~x + PS ~x − ~s||2〈·,·〉= ||~x − PS ~x ||2〈·,·〉 + ||PS ~x − ~s||2〈·,·〉
> ||~x − PS ~x ||2〈·,·〉 if ~s 6= PS ~x
![Page 53: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/53.jpg)
Proof
Take any point ~s ∈ S such that ~s 6= PS ~x
||~x − ~s||2〈·,·〉 = ||~x − PS ~x + PS ~x − ~s||2〈·,·〉= ||~x − PS ~x ||2〈·,·〉 + ||PS ~x − ~s||2〈·,·〉> ||~x − PS ~x ||2〈·,·〉 if ~s 6= PS ~x
![Page 54: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/54.jpg)
![Page 55: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/55.jpg)
Denoising
Aim: Estimating a signal from perturbed measurements
If the noise is additive, the data are modeled as the sum of the signal ~xand a perturbation ~z
~y := ~x + ~z
The goal is to estimate ~x from ~y
Assumptions about the signal and noise structure are necessary
![Page 56: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/56.jpg)
Denoising via orthogonal projection
Assumption: Signal is well approximated as belonging to a predefinedsubspace S
Estimate: PS ~y , orthogonal projection of the noisy data onto S
Error:
||~x − PS ~y ||22 = ||PS⊥ ~x ||22 + ||PS ~z ||22
![Page 57: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/57.jpg)
Proof
~x − PS ~y
= ~x − PS ~x − PS ~z= PS⊥ ~x − PS ~z
![Page 58: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/58.jpg)
Proof
~x − PS ~y = ~x − PS ~x − PS ~z
= PS⊥ ~x − PS ~z
![Page 59: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/59.jpg)
Proof
~x − PS ~y = ~x − PS ~x − PS ~z= PS⊥ ~x − PS ~z
![Page 60: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/60.jpg)
Error
error
0
S
PS~y
~y
~x ~z
PS⊥~x
PS~z
![Page 61: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/61.jpg)
Face denoising
Training set: 360 64× 64 images from 40 different subjects (9 each)
Noise: iid Gaussian noise
SNR :=||~x ||2||~z ||2
= 6.67
We model each image as a vector in R4096
![Page 62: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/62.jpg)
Face denoising
We denoise by projecting onto:
I S1: the span of the 9 images from the same subject
I S2: the span of the 360 images in the training set
Test error:
||~x − PS1 ~y ||2||~x ||2
= 0.114
||~x − PS2 ~y ||2||~x ||2
= 0.078
![Page 63: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/63.jpg)
S1
S1 := span
( )
![Page 64: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/64.jpg)
Denoising via projection onto S1
Projectiononto S1
Projectiononto S⊥1
Signal~x
= 0.993 + 0.114
+
Noise~z
= 0.007 + 0.150
=
Data~y
= +
Estimate
![Page 65: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/65.jpg)
S2
S2 := span
(
· · · )
![Page 66: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/66.jpg)
Denoising via projection onto S2
Projectiononto S2
Projectiononto S⊥2
Signal~x
= 0.998 + 0.063
+
Noise~z
= 0.043 + 0.144
=
Data~y
= +
Estimate
![Page 67: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/67.jpg)
PS1 ~z and PS2 ~z
PS1 ~z PS2 ~z
0.007 =||PS1 ~z ||2||~x ||2
<||PS2 ~z ||2||~x ||2
= 0.043
0.0430.007
= 6.14 ≈√
dim (S2)
dim (S1)(not a coincidence)
![Page 68: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/68.jpg)
PS⊥1 ~x and PS⊥2 ~x
PS⊥1 ~x PS⊥2 ~x
0.063 =
∣∣∣∣∣∣PS⊥2 ~x∣∣∣∣∣∣2||~x ||2
<
∣∣∣∣∣∣PS⊥1 ~x∣∣∣∣∣∣2||~x ||2
= 0.190
![Page 69: Vector spaces - NYU Courantcfgranda/pages/OBDA_fall17... · Properties I Forany~x;~y2V,~x+~ybelongstoV I Forany~x2Vandanyscalar , ~x2V I Thereexistsazerovector~0suchthat~x+~0= ~x](https://reader034.fdocuments.us/reader034/viewer/2022042402/5f125f15e2185834bf147d18/html5/thumbnails/69.jpg)
PS1 ~y and PS2 ~y
~x PS1 ~y PS2 ~y