Inner product spaces over R

Definition. Let V be a vector space over R. An inner product on V is a function which associates with

any pair of vectors u,v in V a real number denoted by (uJv) such that the following postulates are

satisfied.

(1) (ulu) > 0 for all u E V and (ulu) = 0 iff u = Q

(2) (ulv) = (vlu) for all u,v E V

(3) (kulv) = k(u1v) for all u,v E V, k E R

(4) (ulv+w) = (ulv) + (ulw) for all u,v,w E V

A vector space over R on which an inner product is defined is called a real inner product w.

Theorem 1 . Simple deductions from the postulates : In an inner product space

(i) (ulkv) = k(ulv)

(ii) (U + vlw) = (ulw) + (vlw)

(iii) (Q(u) = 0

Proof: Exercise.

Example 1. R" with the dot pr0duct.i.e. define (ulv) = u.v

This is called the standard inner product QQ R".

Example 2. V = set of all real valued continuous functions on [a,b] and we define h

b (f~g) = T f(x) dx.

a For f continuous J ( f ( ~ ) ) ~ d z > 0 if ff 0 so that the postulates hold.

a (Note: This kind of inner product space arises in Quantum Mechanics but usually in a complex space.)

Example 3. V = R~ and define ((xl, t2)l(y1, yz)) = xlyl + 2qy2 + 2x2y1 + 5x2y2. This is not the

standard inner product on R~ but it is an inner product. Postulates (2), (3), (4) are easily verified.

For postulate (1) consider

((xi, x2)1(x1, z2)) = X; + 4x1z2 + 5 4 = ( t i + 2 ~ 2 ) ~ + X; > 0 This = 0 iff x, + 2x2 = 0 and x2 = 0 i.e. iff x2 = z1 = 0 as required.

Definition. A finite dimensional real inner product space is called a Euclidean space. Example 1, 3

above are Euclidean spaces. Example 2 is not.

Definition. Lendh (or m) of u = m) = IIuII. We say u is a unit vector iff llull = 1. If v # Q,

a unit vector in the direction of v is L v . llvll

Note: By postulate (I), llull 2 0 and llull = 0 iff u = Q. By postulate (3)

I Theorem 2 . (CauchgSchwarz inequality). In an inner product space over R, I(ulv)I 5 I I u I I I I v I I . I

Proof: If either u or v is the zero vector, both sides are zero and the inequality is true. Hence we can

assume u # Q and v# Q. We will obtain the result by applying postulate (1) to the vector

Using the other postulates expand the RHS to get

hence (ulv) j llullllvll ... (1). This has been shown for any u,v.

Now replace v by -v : - (ulv) 5 llullllvll ... (2).

(1) and (2) imply I(uIv)I 5 llull llvll as required.

Example 4. Show that for f,g continuous on [0,1]

Answer: Apply Cauchy-Schwarz to Example 2 to get this result.

I(UIv)I < I if U,Y # P and there exists 8, 0 5 8 j r, such that Definition. From Theorem 2, - 1 1 ~ 1 1 1 1 ~ 1 1 -

(UIV) B is called the angle between U.V. In R~ or R~ with the dot product, 0 is the usual coso = - 1 1 ~ 1 1 1 1 ~ 1 1 ' angle between u,v.

I Theorem 3. (Triangle inequality). In an inner product space over R Ilu + vll 5 llull + Ilvll. I

ProoJ llu + v1I2 = (u + V I U + v) = (ulu) + 2(ulv) + (vlv)

= llull" +(uIv) + llv1I2

5 (lull2 + 211ullllvll + llv112 by Cauchy-Schwarz

= (Ilull + 11~11)~

Take square roots of both sides to get the required result.

Definition. In an inner product space we say u,v are orthogonal iff (ulv) = 0.

Example 5. In Ft3 with standard inner product, find two mutually orthogonal vectors each of which is

orthogonal to (4, 2, 3).

Answer. Take any vector othogonal to (4, 2, 3) e.g. (0, 3, -2). Now look'for a 3rd vector orthogonal

to both these two. By inspection (-y, 2, 3) will do. Answer: ((0, 3, -2), (7, 2, 3)}.

Noie: Any subspace of an inner product space is also an inner product space with the inner product

defined as in the larger space.

Definition. Let V be an inner product space. Let W, U be subspaces of V, u E V. We say u is

orthogonal to W iff u is orthogonal to every vector in W. We say W is orthogonal to U iff every

vector in W is orthogonal to every vector in U.

Theorem 4. Let u, vl, . . . vr E V (an inner produce space). Then u is orthogonal to L(vl, . . . v,) iff

(uJvi) = 0 for i = 1, . . . r. i.e. u is orthogonal to a finite dimensional space iff u is orthogonal to

each member of a set of generators for this space.

Proof: Exercise.

Theorem 5 . Any set of mutually orthogonal non zero vectors in V (an inner product space)

is linearly independent.

Proof: Exercise : Suppose ul, . . . ur are mutually orthogonal and non zero and suppose

Xlu, + ... Xrur= Q. Take the inner product of both sides with ui to find Xi = 0.

Definiiion. A set of mutually orthogonal unit vectors in an inner product space is called an

orthonormal set of vectors. (By Theorem 5 any orthonormal set of vectors is independent.)

Definiiion. Let ul be a non zero vector in an inner produce space V and let u E V. The vector

(U'U1) u, is called the eomoonent of u in the direction of ul. Note that it is a multiple of ul. (ull.1)

1 1 Let ul, . . . ur be a mutually orthogonal set of non zero vectors in V and let u E V. - 1

Let v = u - C (component of u in the direction of ui), then v is orthogonal to ul, . . . ur. i = l

Theorem 6. Any Euclidean vector space has an orthonormal basis.

Proof: Let V be a Euclidean vector space. The proof is by induction on the dimension of V. If V

has dimension 1 any unit vector in V will constitute an orthonormal basis. Suppose the result is true

for spaces of dimension n and let V have dimension n+l. Let W be a subspace of V of dimension n.

By the induction hypothesis W has an orthonormal basis {ul; ... un). Extend this to a basis n

{u,, . . . un, u) for V. Let v = u - (component of u in direction of ui) i = l

(1)

Then

(i) v # 4 because if v = 0, (1) would be a dependence relation between u,, . . . un, u.

(ii) v is orthogonal to u,, . . . un by the previous lemma.

Now let u ~ + ~ = LV = unit vector in direction of v. We then have that {u,, . . . un, u ~ + ~ ) is an 1 1 ~ 1 1

orthonormal set in V with n + 1 elements. From Theorem 5, this set is independent and hence a

basis for \I. This completes the proof.

Given a basis {vl, . . . vn) for V, there is a method called the Gram-Schmidt orthogonalization process

whereby this basis can be converted to an orthonormal basis. It amounts to repeated application of the

Lemma p. 40. Vectors u,, . . . un are defined successively by the equations:

U1 = v1

u2 = v2 - component of v2 in direction of u1

u3 = v3 - component of v3 in direction of ul - component of v3 in direction of u2

n- 1 Un = Vn - . C (component of vn in direction of ui)

1 = 1

It is easily shown that for i = 1, 2, . . . n; L(ul, . . . ui) = L(vl, . . . vi), ui # 0 and ui is orthogonal

to u,, u,, . . . U. Hence ul, . . . un are non zero mutually orthogonal vectors. 1-1' u.

If we set w. = 2 , {wl, . . . wn) is an orthonormal basis for V. lluill

Example 6. In R~ let vl = (1, 2, 1, I), v2 = (1, 2, 0, I), v3 = ( 1, -1, 1, 0). Verify that

v,, v,, v3 are independent and use the Gram-Schmidt process on {v,, v2, v3) to find an orthonormal

basisfor V = L(vl,v2,v3).

Answer. ul = vl = (1, 2, 1, 1)

An orthonormal basis is

{* ( 1 1 21 1, 1)i (1, 21-61 11, (79-41 01 1) } 642 6 6

Definition. A square matrix A over W is said to be orthoeonal iff A is non-singular and A-' = A ~ .

Example 7. Verify that the matrix -a 4 I is orthogonal.

Theorem 7 . Let A be a real square n x n matrix. The following conditions are equivalent:

(i) A is orthogonal.

(ii) The rows of A form an orthonormal set in Rn (with standard inner product)

(iii) The columns of A form an orthonormal set in Rn (iv) ATA = I ( v) A A ~ = I

Proof: Suppose A is orthogonal then AAT = I and Ai . A- (i , j) th entry of I J - 0 if i# j - { 1 if i=j = Sij (defn)

This shows (i) implies (ii).

Suppose the rows of A form an orthonormal set then Ai- A. = 6.. and AAT = I. By Theorem J 'J

7(vii) of matrices, A ~ = A-'. This shows (ii) implies (i).

Similarly, using ATA = I we can show the equivalence of (i) and (iii). The equivalence of (i), (iv),

(v) is an immediate consequence of Theorem 7 (vii) of matrices.

One of the reasons orthonormal bases are useful is that it is very easy to express a vector as a linear

combination of the elements of an orthonormal basis.

Theorem 8. Let ul, . . . un be an orthonormal basis for V and let u E V. Then n

u = (ulul)ul + (uIu2)u2 + -.. + (ulun)un = C (component of u in direction of ui) i = l

Proof: We know u = Alul + A2u2 + -. - Anun for some Ai E R.

Hence (ului) = Al(ullui) + A2(u21ui) + + An(unlui).

The only non zero term on the RHS is Ai(uilui) = Ai which is the required result.

Example 8. Let ul, u,, u3 be the rows of the matrix in Example 7. Say why {u,, u,, u3) is a basis for

R~ and find the co-ordinate vector of (1, 2, 3) in this basis.

Answer. By Theorem 7, {ul, u,, u3) is an orthogonal set in R ~ . Hence it is independent (Theorem 5)

and thus a basis for R ~ .

((1, 2, 3)lu1) = 4(2 + 4 + 3) = 3

((1, 2, 3)ln2) = $(-2 + 2 + 6) = 2

((1, 2, 3)lu3) = $( 1 - 4 + 6) = 1

Hence (1, 2, 3) = 3ul + 2u2 + u3.

Co-ordinate vector = (3, 2, 1).

We now consider orthonormal bases and their relationship to orthogonal axes in R3.

* Let OXY be standard axes in R, in the directions of el, e,.

Y' Let {ul, u,) be an ordered orthonormal set in R2.

We can construct a new pair of I r axes in the directions of ul, u2 called OX'Y'. We have shown two

examples of this. In figure (i) OX'Y' have the same "orientationn as OXY (or we may say {ul, u,}

has the same orientation as {el, e2)). In figure (ii) OX'YI have opposite orientation to OXY.

Take any P E R,. The co-ordinates of P w.r.t OXY are the co-ordinates of P w.r.t the basis {el, e,)

i.e. p = (x,y) = m1 + ye,. The co-ordinates of P w.r.t. OX'Y' are the co-ordinates of P w.r.t. the

basis {ul, u2). Denoting the co-ordinates of P w.r.t OX'Y' by (XI, y1 ) we have P = xlul + y1u2 or I

(2, y) = x1u1 + y u,. Dot both sides with ul, u2: I

(x, Y) ' UI = X and (x, y) u2 = y I

(1) - Inmatrixform[xl yl] = [x y ]H2X2 whereH - ul, H ( ~ ) = u2. (1)

The columns of H form the orthonormal set {u,, u2) so H is orthogonal. Transposing (1) we get

K is also orthogonal, the rows of K are ul, u2. Premultiplying (2) by K-' = H yields [ ! ]=H[:i]. These are the equations of the reverse transformation expressing x,y in terms of zl, yl.

cos 0 -sin 8 In figure (i) H =

sin 0 -cos 0 [ c 0 s 8 sin"]-

In figure (i) det H = cos28 + sin20 = 1. In (ii) det II = -(cos28 + sin20) = -1.

Example 9. Show that the equation

14x2 - 4xy + lly2 = 5 represents an

ellipse with one axis in the direction

(192). X

Answer. Take axes 0x1Y1 as shown. Then ul = G(l, 217 U, = -(-2, 1 1). 6

Hence the equation becomes XI)^ + 3(y1)2 = 1 which is the standard form for an ellipse.

Note: The axes OXIYI are called princi~al axes for the ellipse and later we develop a method for

finding them (Theorems 14, 15).

Theorem 9.

(a) If H is an orthogonal matrix then det H = f 1.

(b) The product of two orthogonal matrices is an orthogonal matrix.

Proof:

(a) HHT = I therefore det H det HT = det I. Since det HT = det H and det I = 1 it follows that

(det H ) ~ = 1 and hence det H = f 1. T T

(b) If H, K are orthogonal then ( H K ) ~ ( H K ) = K H HK = KTIK = KTK = I as required.

Consider an ordered orthonormal set {ul, u2, u3) in W3. Let OXYZ be standard axes in Ft3 and let 1 1 1 OX Y Z be axes pointing in the directions of ul, u2, u3 respectively. Take P = (x, y, z) E W3. Let

1 1 1 I I (z', yl, zl) be the ceordinates of P w.r.t to OX Y Z axes. Then (x, y, z) = x ul+ y u2+ z1u3. As in

W 2 dot this with u,, u2, u3 to get [rl y1 el] = [z y z] H where d i ) = ui i. e. the columns of H are the

vectors ul, u2, us. This implies that H is orthogonal. As in we find

[;] = H T [ ] and [ k ] = I I [ ; ] -

I I 1 Definition. If det H = +1 we say the axes OX Y Z have the same orientation as OXYZ.

If det H = -1 we say the axes OXIYIZ' have the owosite orientation as OXYZ.

Note: We will not prove it here, but axes with the same orientation can be superimposed (in the

correct order and direction) by a rigid body movement (rotation) of space while those with opposite

orientation cannot. If one of the vectors ul, u2, u3 is reversed in direction, the orientation of the

corresponding axes changes. It is generally true (and easily shown) that in changing from one

orthonormal basis to another in an Euclidean space, the transition matrix ( ie . B) is an orthogonal

matrix.

Definition. Let W be a subspace of an inner product space V. Define W' = {u : u E V and u is

orthogonal to all elements of W). eg. in R~ let S be a plane through the origin and let L be the line

through the origin normal to S. Then L = S' and S = L'.

Theorem 11.

Let W be a subspace of an inner product space V, then W' is a vector subspace of V and

w n W' = {Q).

Proof: Take u, v E W' and take any w E W. Then (ulw) = 0 and (vlw) = 0 hence (u+vlw) = 0.

Hence u + v E w'. Also (kul w) = k(u1w) = 0 so ku E w I . This shows W' is a subspace of V.

Take u E W n W' then u E W and u E W' hence (ulu) = 0. By postulate (I), u = Q

showing W n W' = (0).

Theorem 12. Let V be a Euclidean space and W a subspace of V, then

(i) each v E V is uniquely expressible in the form v = w + u with w E W and u E W I

(ii) dimension W + dimension W' = dimension V

(iii) (w')' = W

Proof:

(i) We first show V is expressible in the stated form. Let {ul, ... ur) be an orthogonal basis

for W. Let w = (component of v in direction of ui), then W E W. Let u= v-w. By Lemma

page 40 , u E W" and we have v = u + w. For uniqueness: Suppose v = w + u and v = wl+ u' I I I subtract : Q = (w - w ) + (u - u ) , i.e. w -w = u - u'.

I I This shows w' - w E W n W' = {Q), hence w - w = 0 = u - u . (ii) Exercise. Make use of (i) and Theorem 11 to show that (a basis for W) U (a basis for w') =

(a basis for V).

(iii) w E W w orthogonal to all vectors in W I

I I * w E ( W ) . I I I I This shows W C (w')'. From (ii), W and (W ) both have the same dimension so W = (W ) .

Corollary: Let Amxn be a matrix over R. Let X be a column vector. AX = Q iff X is orthogonal

(in $2" with dot product) to each row of A. Hence (solution space of AX = Q) = (row space of A) I

in Rn. By Theorem 12 (ii) (dimension row space of A) + (dimension solution space of AX = 0) =

dimension R" = n. Thus Theorem 12 provides an alternative proof of this result.

We will now show (Theorem 14) that a symmetric matrix is diagonalizable. We have to show the

existence of a basis of eigenvectors. It turns out that an orthonormal basis of eigenvectors can be

chosen.

Theorem 13. Let Anxn be a real symmetric matrix. Suppose A has two different eigenvalues

A,, A, then the eigenspaces corresponding to A,, A, are mutually orthogonal in R"

(with the standard inner product).

Proof: Let Aul = Alul and Au2 = A2u2 with A, # A,. It will suffice to show ul u, = 0.

Let u,, u2 be written as n x 1 matrices and consider T

U, Au,. This 1 x 1 matrix can be written in two ways : T T UTAU, = (Au,) u1 = A2u2 u1 = A 2 [ ~ ~ . ~ 1 ] 1 ~ 1 , and

U ~ A U , = u ~ ( A u , ) = u a ( ~ ~ u l ) = ~ , u T u l = Al[uz.ulllxl

Since A, # A2 we must have uZSu1 = 0.

Theorem 14. Let Anxn be a real symmetric matrix. Then there exists an orthogonal matrix H such

that T H AH = [' . ] (diagonal). Each column of

An

H is an eigenvector of A and Ai is the eigenvalue corresponding to H(~). The columns of H form

a basis of eigenvectors of A.

Proof: We will show the existence of H by induction on n. When n = 1 we can take H = Ilx,. The

theorem is thus true when n = 1. Suppose H exists for n x n symmetric matrices , let A be an (n + 1) x

(n + 1) symmetric matrix. A has a t least one eigenvalue A (corollary to Theorem 16 of Matrices over

64). Let Au = Xu with u a unit vector. Let W = L(u). W is a 1-dimensional subspace of R"+' so I W is an n-dimensional subspace of R"'~. Let {ul, ... un) be an orthonormal basis for &. Then

{ , u . . , } is an orthonormal set. Let K be a matrix with K(')= u, K ( ~ ) = u,, . . ., K(n+l) = Un.

Then K is orthogonal because its columns form an orthonormal set.

A 0 0 - a .

Also T (1)

[To show this : element (1,l) of K ~ A K = K(')-AK(') = u.(Au) = A; element (2,l) of K ~ A K =

K(*).AK(') = U,.AU = AU,.U = O.

Similarly elements (3,1), . .. (n+l, 1) of K ~ A K are zero. K ~ A K is symmetric because ( K ~ A K ) ~ =

= KTAK. Hence elements (1,2), (1,3) . . . (1, n+l) are zero.] K A ( K )

Since KTAK is symmetric, B is symmetric so by the induction hypothesis 3 Lnxn orthogonal so that

its rows (or columns) form an orthonormal set. Premultiply (1) by M~ and post multiply by M.

0 T T M K AKM = L~

I I i

This shows the existence of H.

Suppose we have H orthogonal with H ~ A H = [' . 1. An

Premultiply by H : AH = H (since H H ~ = I).

(0 Equating columns of each side : AH(^) = XiH . This shows that Xi is an eigenvalue corresponding to H(~? Since an nxn orthogonal matrix is non-

singular its columns will be independent and thus form a basis for Rn.

Ezercise 10. Let A = 2 2 2 1 0 2 1 4

Show that - 1 is an eigenvalue for A. Find an

orthogonal H so that H ~ A H is diagonal.

Answer, A -(-1)1 = A + I = I : : I and det (A + I) = 2(12 - 4) - 2.8 = 0.

L -I Hence - 1 is an eigenvalue.

The columns of H are eigenvectors of A. First find the eigenvalues.