5

1

Signal , Weight Vector Spaces and Linear Transformations

5

2

Notation

x

x1

x2

xn

=

Vectors in Ân. Generalized Vectors.

n+

𝐱

𝐱=[𝑎1 ,1 𝑎 1,2𝑎2 ,1 𝑎 2 ,2]

5

3

Vector Space

1. An operation called vector addition is defined such that if x Î X and y Î X then x + y Î X.

2. x + y = y + x3. (x + y) + z = x + (y + z)

4. There is a unique vector 0 Î X, called the zero vector, such that x + 0 = x for all x Î X.

5. For each vector there is a unique vector in X, to be called (-x ), such that x + (-x ) = 0 .

5

4

Vector Space (Cont.)

6. An operation, called multiplication, is defined such that for all scalars a Î F, and all vectors x  Î X, a x  Î X.

7. For any x  Î X , 1x = x (for scalar 1).8. For any two scalars a Î F and b Î F, and any x Î X,

a (bx) = (a b) x .9. (a + b) x = a x + b x .10. a (x + y)  = a x + a y

5

5

Examples (Decision Boundaries)

Is the p2, p3 plane a vector space?

5

6

Examples (Decision Boundaries)

Is the line p1 + 2p2 - 2 = 0 a vector space?

5

7

Other Vector Spaces

Polynomials of degree 2 or less.

x 2 t 4t2+ +=

y 1 5 t+=

Continuous functions in the interval [0,1].:

5

8

Linear Independence

a1x1 a2 x2 anxn+ + + 0=If

implies that each ai 0=

then xi is a set of linearly independent vectors.

5

9

Example

p1

1–11–

= p2

111–

=

a1p1 a2 p2+ 0=

Let

a– 1 a2+

a1 a2+

a– 1 a2– +

000

=

This can only be true if

a1 a2 0= =

Therefore the vectors are independent.

5

10

Problem P5.3(Gramian)

i.

5

11

Problem P5.3(Gramian)(cont.)

- =0==4

5

12

Gramian

𝐏=[𝐱 1 , 𝐱 2 ,⋯ ,𝐱𝑛 ]

G (𝐱 1 ,𝐱 2 ,⋯ ,𝐱 𝑛)=|𝐏T ×𝐏|=|[ (𝐱 1 , 𝐱 1) (𝐱 1, 𝐱 2)(𝐱 2 , 𝐱 1) (𝐱 2 ,𝐱 2)

⋯ (𝐱 1 , 𝐱𝑛)(𝐱 2 , 𝐱𝑛)

⋮ ⋱ ⋮(𝐱𝑛 , 𝐱 1) (𝐱𝑛 ,𝐱 2) ⋯ (𝐱𝑛 , 𝐱𝑛)]|

0 iff are linearly independent

0 iff are linearly dependent

5

13

Spanning a Space

A subset spans a space if every vector in the space can be written as a linear combination of the vectors in the subspace.

x x1u1 x 2u2 x mum+ + +=

5

14

Basis Vectors

• A set of basis vectors for the space X is a set of vectors which spans X and is linearly independent.

• The dimension of a vector space, Dim(X), is equal to the number of vectors in the basis set.

• Let X be a finite dimensional vector space, then every basis set of X has the same number of elements.

5

15

Example

Polynomials of degree 2 or less.

u1 1= u2 t= u3 t2=

(Any three linearly independent vectors in the space will work.)

u1 1 t–= u2 1 t+= u3 1 t t+ +2

=

Basis A:

Basis B:

How can you represent the vector x = 1+2t using both basis sets?

𝐮 1+𝐮2=2 ,𝐮 2 −𝐮 1=2 t =

5

16

Inner Product / Norm

A scalar function of vectors x and y can be defined asan inner product, (x , y ), provided the following aresatisfied (for real inner products):1. (x , y ) = (y , x ) .2. (x , ay1+by2) = a( x , y1) + b( x , y2) .3. (x , x) 0 , where equality holds iff ≧ x = 0 .

A scalar function of a vector x is called a norm, ||x || provided the following are satisfied:1. ||x || 0 .≧2. ||x || = 0 iff x = 0 .3. ||a x || = |a| ||x || for scalar a .4. ||x + y || ||x || + ||y || .

5

17

Example

xTy x1 y1 x 2y2 x nyn+ + +=

Standard Euclidean Inner Product

Standard Euclidean Norm

Angle:

||x || = (x , x)1/2 ||x|| = (xTx)1/2 =

cosq =

(𝑥 , 𝑦 )=∫0

1

𝑥 (𝑡 ) 𝑦 (𝑡 )𝑑𝑡

Inner Product(see Problem P5.6):

5

18

Problem P5.6

An inner product must satisfy the following properties.1. (𝑥 , 𝑦 )=(𝑦 ,𝑥)

(𝑥 , 𝑦 )=∫0

1

𝑥 (𝑡 ) 𝑦 (𝑡 )𝑑𝑡=∫0

1

𝑦 (𝑡 )𝑥 (𝑡 )𝑑𝑡=(𝑦 , 𝑥)

2. (𝑥 ,𝑎𝑦 1+𝑏𝑦 2 )=𝑎 (𝑥 , 𝑦 1 )+𝑏(𝑥 , 𝑦 2)

5

19

Problem P5.6(cont.)

3. (𝑥 ,𝑥 ) ≥ 0 , where equality holdsiff x isthe zero vector

(𝑥 , 𝑥 )=∫0

1

𝑥 (𝑡 )𝑥 (𝑡 )𝑑𝑡∫0

1

𝑥 (𝑡 )2𝑑𝑡≥ 0

,which is the zero vector

5

20

Orthogonality

Two vectors x , y ÎX are orthogonal if (x , y) = 0 .

Any vector in the p2,p3 plane isorthogonal to the weight vector.

Example

5

21

Gram-Schmidt Orthogonalization

Independent Vectors Orthogonal Vectors

Step 1: Set first orthogonal vector to first independent vector.

Step 2: Subtract the portion of y2 that is in the direction of v1.

Where a is chosen so that v2 is orthogonal to v1:

𝑦 1 , 𝑦 2 , …, 𝑦𝑛 𝑣 1 ,𝑣 2 ,…,𝑣𝑛

𝑣 1=𝑦1

𝑣 2=𝑦 2−𝑎𝑣 1

(𝑣 1 ,𝑣 2 )=(𝑣1 , 𝑦 2 −𝑎𝑣 1 )=(𝑣 1 , 𝑦2 )−𝑎 (𝑣 1,𝑣 1 )=0

𝑎=(𝑣1 , 𝑦 2)(𝑣1 ,𝑣1)

5

22

Gram-Schmidt (Cont.)

Projection of y2 on v1:

Step k: Subtract the portion of yk that is in the direction of allprevious vi .

(𝑣1 , 𝑦 2)(𝑣 1 ,𝑣1)

𝑣1

𝑣 𝑘=𝑦𝑘−∑𝑖=1

𝑘−1 (𝑣𝑖 , 𝑦𝑘)(𝑣𝑖 ,𝑣𝑖) 𝑣𝑖

5

23

Example

=.

5

24

Example

Step 1： 𝐯 1=𝐲 1=[21]

5

25

Example (Cont.)

Step 2.

5

26

Vector ExpansionIf a vector space X has a basis set {v1, v2, ..., vn},

then any x ÎX has a unique vector expansion:

If the basis vectors are orthogonal, and wetake the inner product of vj and x :

Therefore the coefficients of the expansion can be computed:

(𝑣 j , 𝜒 )=(𝑣 j ,∑𝑖=1

𝑛

𝑥 i𝑣 i)=∑𝑖=1

𝑛

𝑥 i(𝑣 j ,𝑣 i)=𝑥 j(𝑣 j ,𝑣 j)

𝑥 j= (𝑣 j , 𝜒 )(𝑣 j ,𝑣 j )

5

27

Column of Numbers

The vector expansion provides a meaning forwriting a vector as a column of numbers.

To interpret x, we need to know what basis was usedfor the expansion.

𝑥=[ 𝑥1𝑥 2⋮𝑥 𝑛]

5

28

Reciprocal Basis VectorsDefinition of reciprocal basis vectors, ri:

where the basis vectors are {v1, v2, ..., vn}, andthe reciprocal basis vectors are {r1, r2, ..., rn}.

RT B I=

B v1 v2 vn= R r1 r2 rn=

For vectors in Ân we can use the following inner product:

Therefore, the equations for the reciprocal basis vectors become:

RT B 1–=

(𝑟 𝑖 ,𝑣 𝑗 )=𝑟 𝑖𝑇 𝑣 𝑗

5

29

Vector Expansion

Take the inner product of the first reciprocal basis vectorwith the vector to be expanded:

By definition of the reciprocal basis vectors:

Therefore, the first coefficient in the expansion is:

In general, we then have (even for nonorthogonal basis vectors):

(𝑟 1 ,𝑣2 )=(𝑟 1 ,𝑣3 )=…=(𝑟 1 ,𝑣𝑛)=0(𝑟 1 ,𝑣1 )=1

𝑥1=(𝑟 1 , 𝜒 )

𝑥 𝑗=(𝑟𝑗 , 𝜒 )

5

30

Example

Basis Vectors:

Vector to Expand:

𝑣 1𝑠=[21] ,𝑣2 𝑠=[12]

𝑥 𝑠=[ 032 ]

5

31

Example (Cont.)

Reciprocal Basis Vectors:

Expansion Coefficients: Matrix Form:

𝐑 T=[2 11 2 ]−1=[ 2

3 − 13

− 13

23 ]=[𝐫 1𝑡

𝐫 2𝑡 ]

𝐫 1=[ 23

− 13 ]𝐫 2=[− 1

323 ]

𝐱 1𝑣=𝐫 1𝑡 𝐱 𝑠=[ 23 − 13 ][ 0

32 ]=− 1

2

𝐱 2𝑣=𝐫 2 𝑡 𝐱 𝑠=[− 13

23 ] [ 032 ]=1

5

32

Example (Cont.)

The interpretation of the column of numbersdepends on the basis set used for the expansion.

χ=0 𝑠1+ 32 𝑠2=− 1

2 𝑣1+1𝑣 2

5

33

Linear Transformation

= =

=

==

𝐱𝑣=𝐁−1 𝐱 𝑠 .This operation , call a change of basis