VECTOR SPACE INTERPRETATION OF RANDOM VARIABLES

Interpretation of random variables as elements of a vector space helps in understanding many

operations involving random variables. We start with an introduction to the concepts of the

vector space. We will discuss the principles of the minimum mean-square error estimation and

the linear minimum mean-square error estimation of a signal from noise and the vector-space

interpretation of the later.

Vector space

Consider a set V with elements called vectors and the set of real numbers . The elements of

will be called scalars. A vector will be denoted by bold-face character like v . Suppose two

operations called vector addition and scalar multiplication respectively on V.

V is called a vector space if the following properties are satisfied.

(1) ( V ,+) is a commutative group. Thus ( V ,+) satisfies the following properties.

(i) Closure property: For any pair of elements ,v,w V there exists a unique

element .(v + w) V

(ii) Associativity: Vector addition is associative: v +(w + z) = (v + w) + z for

any three vectors v,w,z V

(iii) Existence of the zero vector: There is a vector 0 V such that

v +0 = 0+ v v for any .v V

(iv) Existence of the additive inverse: For any v V there is a vector v V such

that

.v +(-v) = 0 = (-v) + v

(v) Commutativity: For any ,v,w V .v + w = w + v

(2) For any element v V and any .r the scalar multiplication r v V

This the scalar multiplication has the following properties for any , .r s and any

,v,w V

(1) Associativity: ( ) ( ) for , and r s rs r s v v v V

(2) Distributivity with respect to vector addition: r r r(v + w) = v + w

(3) Distributivity with respect to sacalar addition:

r s r s( )v = v + v

(4) Unity scalar: 1v = v

Example 1:

Let S be an arbitrary set and V be the set of all functions from S to . Suppose :f S and

:g S denote two functions and s S .and a . Then by definition,

( )( ) ( ) ( )f g s f s g s and

( )( ) ( )af s af s .

Therefore, addition of functions and scalar multiplication of a function are function in V. It is

easy to verify that addition of functions and the multiplication of a function by scalar satisfies

the properties of vector space. Particularly, the zero function is a function that maps all

elements in S to the real number 0. Thus,

0( ) 0,s s S

The random variables defined on a probability space ( , , )S P are functions on the sample space.

Therefore, the set of random variables forms a vector space with respect to the addition of

random variables and scalar multiplication of a random variable by a real number.

Subspace

Suppose W is a non-empty subset of V. W is called a subspace of V if W is a vector space

with respect to the vector addition and scalar multiplication defined on V.

For a non-empty subset W is a subspace of , it is sufficient that W is closed under the vector

addition and the scalar multiplication of V. Thus the sufficient conditions are:

(1) , , and

(2) , ,r r

v w W (v + w) W

v W v W

Linear Independence and Basis

Consider a subset of n vectors 1 2 nB = {b ,b , ...,b }

If 1 2 .... nc c c 1 2 n

b b b 0 implies that

1 2 ... 0,nc c c then n1 2b ,b , ...,b are called linearly independen (LI)t.

The subset 1 2 n

B = {b ,b , ...,b } of n LI vectors is called a basis if each v V can be expressed

as a linear combination of elements of B. The number of elements in B is called the dimension

of V. Thus B = {i, j,k} is a basis of 3 .

Norm of a vector

Suppose v is a vector in a vector space V defined over . The norm, v is a scalar such that

and V r v,w

1. 0

2. 0 only when

3.

4. ( Triangle Inequality )

r r

v

v v 0

v v

v w v w

A vector space V where norm is defined is called an normed vector space. For example,

following are valid norms of 1 2[ ... ] n

nv v v v :

(i) 2 2 2

1 2 ... nv v v v

(ii) 1 2max( , ,..., )nv v vv

Inner Product

If v and w are real vectors in a vector space V defined over , the inner product v,w is a

scalar such that and V r v,w,z

2

1.

2. 0, where is a induced by the inner product

3.

4.

norm

,

r , r

v, w w, v

v, v v v

v w z v, z w, z

v w v, w

A vector space V where an inner product is defined is called an inner product space. Following

are examples of inner product spaces:

(i) n with the inner product v, w =

1 1 2 2 1 2 1 2... , [ ... ] , [ ... ]n n

n n n nv w v w v w v v v w w w v w v w =

(ii) The space 2 ( )L of square-integrable real functions with the inner product

1 2 1 2, ( ) ( )f f f x f x dx

2

1 2, ( )f f L

Cauchy Schwarz Inequality

For any two vectors v and w belonging to an inner product space V ,

v, w w v

Let us consider

2

2 22

then , 0

, 2 , , 0

2 , 0

c c c

c c c

c c

z v w z v w v w

v v v w w w

v v w w

The left-hand side of the last inequality is a quadratic expression in variable c For the above

quadratic expression to be non-negative, the discriminant must be non-positive.. So

2 22

2 22

2 22

| 2 , | 4 0

| , | 0

,| |

| , |

v w v w

v w v w

wv v w

v w v w

The equality holds when

0 0c

c

z v w

v w

Hilbert space

Note that the inner product induces a norm that measures the size of a vector. Thus, v w is a

measure of distance between the vectors ,v,w V We can define the convergence of a

sequence of vectors in terms of this norm.

Consider a sequence of vectors , 1,2,...,n n

v The sequence is said to converge to a limit

vector v if corresponding to every 0 , we can find a positive integer N such that

for .n N nv v The sequence of vectors , 1,2,...,n n

v is said to be a

Cauchy sequence if ,

lim 0 .mn m

nv v

In analysis, we may require that the limit of a Cauchy sequence of vectors is also a member of

the inner product space. Such an inner prduct space where every Cauchy sequence of vectors is

convergent is known as a Hilbert space.

Orthogonal vectors

Two vectors v and w belonging to an inner product space V are called orthogonal if

0 v, w

Orthogonal vectors are independent and a set of n orthogonal vectors 1 2 nB = {b ,b , ...,b } forms a

basis of the n-dimensional vector space.

Orthogonal projection

It is one of the important concepts in linear algebra widely used in random signal processing.

Suppose W is a subspace of an inner product space V. Then the subset

{ | , 0 }v v W V v,w w W

is called the orthogonal complement of W.

Any vector v in a Hilbert space V can be expressed as

1v = w + w

where 1, w W w W . In such a decomposition, w is called the orthogonal projection of v

on W and represents closest approximation of v by a vector in W in the following sense

min

u W

v w v u

We omit the proof of this result. The result can be geometrically illustrated as follows:

Gram-Schimdt orthogonalisation

Joint Expectation as an inner product

Interpreting the random variables and X Y as vectors, the joint expectation EXY satisfies the

properties of the inner product of two vectors. and .X Y Thus

< > XYX, Y = E

We can also define the norm of the random variable X by

2 2< > XX X, X = E

Similarly, two random variables and X Y are orthogonal if < > XYX, Y = E = 0

We can easily verify that EXY satisfies the axioms of inner product.

The norm of a random variable X is given by

2 2X = EX

u 1w

v

v

wv

Wv

For two n dimensional random vectors

1

2

n

X

X

X

X and

1

2,

n

Y

Y

Y

Y the inner product is

1

n

i ii

< > = E EX Y

X, Y X Y

The norm of an n-dimensional random vector X is given by 2 2

1

n

ii

< > = E EX

X X, X X X

Orthogonal Random Variables and Orthogonal Random Vectors

Two vectors v and w are called orthogonal if 0 v, w

Two random variables YX and are called orthogonal if 0.EXY

Similarly two n-dimension random vectors and X Y are called orthogonal if

1

0n

i ii

E EX Y

X Y

Just like the independent random variables and the uncorrelated random variables, the

orthogonal random variables form an important class of random variables.

If YX and are uncorrelated, then

( )( ) 0

( ) is orthogonal to ( )

X Y

X Y

E X Y

X Y

If each of YX and is of zero-mean

( , )Cov X Y EXY

In this case, 0 ( ) 0.EXY Cov XY