Stats 443.3 & 851.3

Linear Models

Instructor: W.H.Laverty

Office: 235 McLean Hall

Phone: 966-6096

Lectures:M W F

9:30am - 10:20am Geol 269Lab 2:30pm – 3:30 pm Tuesday

Evaluation: Assignments, Term tests - 40%Final Examination - 60%

• The lectures will be given in Power Point

Course Outline

Introduction

Review of Linear Algebra and Matrix Analysis

Review of Probability Theory and Statistical

Theory

Multivariate Normal distribution

The General Linear ModelTheory and Application

Special applications of The General Linear Model

Analysis of Variance Models, Analysis of Covariance models

Independent variables

Dependent Variables

Categorical Continuous Continuous & Categorical

Categorical Multiway frequency Analysis(Log Linear Model)

Discriminant Analysis Discriminant Analysis

Continuous ANOVA (single dep var)MANOVA (Mult dep var)

MULTIPLE REGRESSION(single dep variable)MULTIVARIATEMULTIPLE REGRESSION (multiple dependent variable)

ANACOVA (single dep var)MANACOVA (Mult dep var)

Continuous & Categorical

?? ?? ??

A chart illustrating Statistical Procedures

A Review of Linear Algebra

With some Additions

11 12 1

21 22 2

1 2

n

nij

m m mn

a a a

a a aA a

a a a

Matrix AlgebraDefinition

An n × m matrix, A, is a rectangular array of elements

n = # of columns

m = # of rows

dimensions = n × m

1

2

n

v

v

v

v

Definition

A vector, v, of dimension n is an n × 1 matrix rectangular array of elements

vectors will be column vectors (they may also be row vectors)

1

2

n

v

v

v

v

A vector, v, of dimension n

can be thought a point in n dimensional space

v2

v1

v3

1

2

3

v

v

v

v

11 11 12 12 1 1

21 21 22 22 2 2

1 1 2 2

n n

n nij ij

m m m m mn mn

a b a b a b

a b a b a bA B a b

a b a b a b

Matrix OperationsAddition

Let A = (aij) and B = (bij) denote two n × m matrices Then the sum, A + B, is the matrix

The dimensions of A and B are required to be both n × m.

11 12 1

21 22 2

1 2

n

nij

m m mn

ca ca ca

ca ca cacA ca

ca ca ca

Scalar Multiplication

Let A = (aij) denote an n × m matrix and let c be any scalar. Then cA is the matrix

v2

v1

v3

1

2

3

v

v

v

v

Addition for vectors

1

2

3

w

w

w

w

1 1

2 2

3 3

v w

v w

v w

v w

v2

v1

v3

1

2

3

v

v

v

v

Scalar Multiplication for vectors

1

2

3

cv

c cv

cv

v

1

m

il ij jlj

c a b

Matrix multiplication

Let A = (aij) denote an n × m matrix and B = (bjl) denote an m × k matrix

Then the n × k matrix C = (cil) where

is called the product of A and B and is denoted by A∙B

1

m

i ij jj

w a v

In the case that A = (aij) is an n × m matrix and B = v = (vj) is an m × 1 vector

Then w = A∙v = (wi) where

is an n × 1 vector

v2

v1

v3

1

2

3

v

v

v

v

w2

w1

w3

1

2

3

w

w A

w

w v

A

1 0 0

0 1 0

0 0 1

nI I

Definition

An n × n identity matrix, I, is the square matrix

Note:

1. AI = A

2. IA = A.

Definition (The inverse of an n × n matrix)

AB = BA = I,

If the matrix B exists then A is called invertible Also B is called the inverse of A and is denoted by A-1

11 12 1

21 22 2

1 2

n

nij

n n nn

a a a

a a aA a

a a a

Let A denote the n × n matrix

Let B denote an n × n matrix such that

Note: Let A and B be two matrices whose inverse exists. Let C = AB. Then the inverse of the matrix C exists and C-1 = B-1A-1.

Proof

C[B-1A-1] = [AB][B-1A-1] = A[B B-1]A-1 = A[I]A-1

= AA-1=I

The Woodbury Theorem

11 1 1 1 1 1A BCD A A B C DA B DA

where the inverses11 1 1 1, and exist.A C C DA B

Then all we need to show is that

H(A + BCD) = (A + BCD) H = I.

Proof:

Let 11 1 1 1 1H A A B C DA B DA

H A BCD

11 1 1 1 1A A B C DA B DA A BCD

11 1 1 1 1A A A B C DA B DA A

11 1 1 1 1A BCD A B C DA B DA BCD

11 1 1I A B C DA B D

11 1 1 1 1A BCD A B C DA B DA BCD

1I A BCD 11 1 1 1A B C DA B I DA BC D

1I A BCD

11 1 1 1 1A B C DA B C DA B CD

1 1I A BCD A BCD I

The Woodbury theorem can be used to find the inverse of some pattern matrices:

Example: Find the inverse of the n × n matrix

1 0 0 1 1 1

0 1 0 1 1 1

0 0 1 1 1 1

b a a

a b ab a a

a a b

1

11 1 1

1

b a I a A BCD

where1

1

1

B

A b a I 1 1 1D

1 1C a

1 1A I

b a

hence 1 1

Ca

1 1

1

11 11 1 1

1

C DA B Ia b a

and

11 b a nn b a an

a b a a b a a b a

Thus

Now using the Woodbury theorem

11 1

1

a b aC DA B

b a n

11 1 1 1 1 1A BCD A A B C DA B DA

1

11 1 11 1 1

1

1

a b aI I I

b a b a b a n b a

1

111 1 1

1

1

aI

b a b a b a n

Thus

1 0 0 1 1 1

0 1 0 1 1 11

1

0 0 1 1 1 1

a

b a b a b a n

1b a a

a b a

a a b

c d d

d c d

d d c

where

1

ad

b a b a n

1

and 1

ac

b a b a b a n

21 11

1 1

b a na

b a b a n b a b a n

Note: for n = 2

2 2

a ad

b a b a b a

2 2

1and

b bc

b a b a b a

1

2 2

1Thus

b a b a

a b a bb a

Also1

b a a b a a b a a c d d

a b a a b a a b a d c d

a a b a a b a a b d d c

1 ( 2) ( 2)

( 2) 1 ( 2)

( 2) ( 2) 1

bc n ad bd ac n ad bd ac n ad

bd ac n ad bc n ad bd ac n ad

bd ac n ad bd ac n ad bc n ad

Now

1

ad

b a b a n

21and

1

b a nc

b a b a n

22 11

1 1

b a n n abbc n ad

b a b a n b a b a n

22 1

1

b b a n n a

b a b a n

2 2

2 2

2 11

2 1

b ab n n a

b ab n n a

( 2)2( 2)

1 1

b n a ab a nabd ac n ad

b a b a n b a b a n

0

and

This verifies that we have calculated the inverse

11 12

21 22

q

n m n q

p m p

A AA

A A

Block Matrices

Let the n × m matrix

be partitioned into sub-matrices A11, A12, A21, A22,

11 12

21 22

p

m k m p

l k l

B BB

B B

Similarly partition the m × k matrix

11 12 11 12

21 22 21 22

A A B BA B

A A B B

Product of Blocked Matrices

Then

11 11 12 21 11 12 12 22

21 11 22 21 21 12 22 22

A B A B A B A B

A B A B A B A B

11 12

21 22

p

n n n p

p n p

A AA

A A

The Inverse of Blocked Matrices

Let the n × n matrix

be partitioned into sub-matrices A11, A12, A21, A22,

11 12

21 22

p

n n n p

p n p

B BB

B B

Similarly partition the n × n matrix

Suppose that B = A-1

11 12 11 12

21 22 21 22

A A B BA B

A A B B

Product of Blocked Matrices

Then

11 11 12 21 11 12 12 22

21 11 22 21 21 12 22 22

A B A B A B A B

A B A B A B A B

0

0

pp n p

n pn p p

I

I

Hence 11 11 12 21 1A B A B I

11 12 12 22 0 2A B A B

21 11 22 21 0 3A B A B

21 12 22 22 4A B A B I

From (1)1 1

11 12 21 11 11A A B B B

From (3)1 1 1 1

22 21 21 11 21 11 22 210 or A A B B B B A A

Hence 1 111 12 22 21 11A A A A B

using the Woodbury Theorem

or 1111 11 12 22 21B A A A A

11 1 1 1

11 11 12 22 21 11 12 21 11A A A A A A A A A

Similarly11

22 22 21 11 12B A A A A

11 1 1 122 22 21 11 12 22 21 12 22A A A A A A A A A

21 11 22 21 0 3A B A B From

122 21 11 21 0A A B B

11 1 121 22 21 11 22 21 11 12 22 21B A A B A A A A A A

and

11 1 112 11 12 22 11 12 22 21 11 12B A A B A A A A A A

similarly

11 12

21 22

p

n n n p

p n p

A AA

A A

Summarizing

Let

11 12

21 22

p

n p

p n p

B B

B B

Suppose that A-1 = B

then

11 1 121 22 21 11 22 21 11 12 22 21B A A B A A A A A A

11 1 112 11 12 22 11 12 22 21 11 12B A A B A A A A A A

1 11 1 1 1 111 11 12 22 21 11 11 12 22 21 11 12 21 11B A A A A A A A A A A A A A

1 11 1 1 1 1

22 22 21 11 12 22 22 21 11 12 22 21 12 22B A A A A A A A A A A A A A

0 0

0 0

0 0

0 0

p

p

p p

a b

aI bI a bA

cI dI c d

c d

Example

Let

11 12

21 22

p

n p

p n p

B B

B B

Find A-1 = B

11 12 21 22, , ,A aI A bI A cI A dI

1 1111

bc dd d ad bcB aI bI I cI a I I

1 1122

bc aa a ad bcB dI cI I bI d I I

1 121 22 21 11 ( ) d c

d ad bc ad bcB A A B I cI I I

1 112 11 12 22 ( ) a b

a ad bc ad bcB A A B I bI I I

1hence d b

ad bc ad bc

c aad bc ad bc

I IA

I I

11 12 1

21 22 2

1 2

n

nij

m m mn

a a a

a a aA a

a a a

The transpose of a matrixConsider the n × m matrix, A

is called the transpose of A

11 21 1

12 22 2

1 2

m

mji

m m mn

a a a

a a aA a

a a a

then the m × n matrix, (also denoted by AT)A

Symmetric Matrices

• An n × n matrix, A, is said to be symmetric if

Note:

AA

11

111

AA

ABAB

ABAB

The trace and the determinant of a square matrix

11 12 1

21 22 2

1 2

n

nij

n n nn

a a a

a a aA a

a a a

Let A denote then n × n matrix

Then

1

n

iii

tr A a

11 12 1

21 22 2

1 2

det the determinant of

n

n

n n nn

a a a

a a aA A

a a a

also

where1

n

ij ijj

a A

cofactor of ij ijA a

the determinant of the matrix

after deleting row and col.th thi j

11 1211 22 12 21

21 22

deta a

a a a aa a

ji1

1. 1, I tr I n

Some properties

2. , AB A B tr AB tr BA

1 13. A

A

122 11 12 22 2111 12

121 22 11 22 21 11 12

4. A A A A AA A

AA A A A A A A

22 11 12 21 if 0 or 0A A A A

Some additional Linear Algebra

Inner product of vectors

Let denote two p × 1 vectors. Then. and x y

1

1 1 1, , p p p

p

y

x y x x x y x y

y

1

p

i ii

x y

Note:2 21 the length of px x x x x

Let denote two p × 1 vectors. Then. and x y

cos angle between and x y

x yx x y y

x

y

Note:Let denote two p × 1 vectors. Then. and x y

cos angle between and x y

x yx x y y

x

y

0 2 and 0 if yx

.orthogonal are and then ,0 if Thus yxyx

2

Special Types of Matrices

1. Orthogonal matrices– A matrix is orthogonal if PˊP = PPˊ = I– In this cases P-1=Pˊ .– Also the rows (columns) of P have length 1 and

are orthogonal to each other

then P P PP I

Suppose P is an orthogonal matrix

Let denote p × 1 vectors. and x y

Let and u Px v Py

Then u v Px Py x P Py x y

and u u Px Px x P Px x x

Orthogonal transformation preserve length and angles – Rotations about the origin, Reflections

The following matrix P is orthogonal

Example

62

61

61

21

21

31

31

31

0P

Special Types of Matrices(continued)

2. Positive definite matrices– A symmetric matrix, A, is called positive definite

if:

– A symmetric matrix, A, is called positive semi definite if:

022 112211222

111 nnnnn xxaxxaxaxaxAx

0 allfor

x

0 xAx

0 allfor

x

If the matrix A is positive definite then

0 wheresatisfy that , points, ofset the ccxAxx

.0 origin, at the centered

ellipsoid l dimensionaan of surface on the are

n

Theorem The matrix A is positive definite if

0,,0,0,0 321 nAAAA

nnnn

n

n

n

aaa

aaa

aaa

AA

aaa

aaa

aaa

Aaa

aaAaA

21

22212

11211

332313

232212

131211

32212

12112111

and

,,,

where

Example

0421875.0

15.25.125.

5.15.25.

25.5.15.

125.25.5.1

4

AAA

05625.0

15.25.

5.15.

25.5.1

det 33

AA

01 1det ,075.0 15.

5.1det 122

AAA

Special Types of Matrices(continued)

3. Idempotent matrices– A symmetric matrix, E, is called idempotent if:

– Idempotent matrices project vectors onto a linear subspace

EEE

xExEE

xE

x

Example.rank ofmatrix any be Let nmnmA

:Proof

.idempotent is then ,Let -1 EAAAAE

AAAAAAAAEE -1-1

EAAAA

AAAAAAAA

1-

-1-1

Example (continued)

110

011 A

110

011

10

11

01

110

011

10

11

011

1- AAAAE

110

011

10

11

01

110

011

21

12

10

11

01

32

31

31

321

32

31

31

31

32

31

31

31

32

Vector subspaces of n

Let n denote all n-dimensional vectors (n-dimensional Euclidean space).

Let M denote any subset of n.

Then M is a vector subspace of n if:

1. M2.If M and M then M3.If M then M .

0

u

vu

v

u

u

c

Example 1 of vector subspace

Let M

where is any n-dimensional vector.

Then M is a vector subspace of n.

Note: M is an (n - 1)-dimensional plane through the origin.

02211 nnuauaua uau

na

a

a

2

1

a

Proof

Now M 02211 nnuauaua uau

.0 1. 0a

0 n the

0 and 0 If 2.

vauavua

vaua

0n the

0 If 3.

uaua

ua

cc

0 since

any vector toorthogonal is vector the

ua

uaNote

Projection onto M.Let be any vector

M is

plane thelar toperpendicu

through line theofequation The

axu

x

t

x

aaa

xaxaxu

aa

xaaaxaua

u

axu

t

tt

t

t

proj and

and 0 i.e.

plane. on the is that sochosen is if

plane theonto projection theis point The

Example 2 of vector subspace

Let M

Then M is a vector subspace of n.

M is called the vector space spanned by the p

n -dimensional vectors:

M is a the plane of smallest dimension through the origin that contains the vectors:

ppbbb aaauu

2211

vectors.ldimensiona ofset any is ,,, 21 nppaaa

paaa

,,, 21

paaa

,,, 21

Eigenvectors, Eigenvalues of a matrix

Definition

Let A be an n × n matrix

Let and be such thatx

with 0Ax x x

then is called an eigenvalue of A and

and is called an eigenvector of A andx

Note:

0A I x

1If 0 then 0 0A I x A I

thus 0 A I

is the condition for an eigenvalue.

11 1

1

det = 0n

n nn

a a

A I

a a

= polynomial of degree n in .

Hence there are n possible eigenvalues 1, … , n

0 if 0x Ax x

Proof A is positive definite if

be an eigenvalue and

Thereom If the matrix A is symmetric then the eigenvalues of A, 1, … , n,are real.

Thereom If the matrix A is positive definite then the eigenvalues of A, 1, … , n, are positive.

and x Let

corresponding eigenvector of A.

then Ax x

and , or 0x x

x Ax x xx Ax

Proof: Note

Thereom If the matrix A is symmetric and the eigenvalues of A are 1, … , n, with corresponding eigenvectors

i.e. i i iAx x 1, , nx x

If i ≠ j then 0 i jx x

j i i j ix Ax x x

and i j j i jx Ax x x

0 i j i jx x

hence 0 i jx x

Thereom If the matrix A is symmetric with distinct eigenvalues, 1, … , n, with corresponding eigenvectors

1 1 1then n n nA x x x x

1, , nx x

Assume 1 i ix x

1 1

1

0

, ,

0n

n n

x

x x

x

PDP

proof

Note 1 i ix x

1 1 1 1

1

1

, ,n

n

n n n n

x x x x x

P P x x

x x x x x

and 0 if i jx x i j

1 0

0 1

I

P is called an orthogonal matrix

therefore

1

1 1 1, , n n n

n

x

I PP x x x x x x

x

thus

1 1 and .P P PP PP I

1now i iAx x

1 1 1 1 1 n n n n nAx x Ax x x x x x

and i i i i iAx x x x

1 1 1 1 1 n n n n nA x x x x x x x x

1 1 1 n n nA x x x x

Comment

The previous result is also true if the eigenvalues are not distinct.

Namely if the matrix A is symmetric with eigenvalues, 1, … , n, with corresponding

eigenvectors of unit length

1 1 1then n n nA x x x x

1, , nx x

1 1

1

0

, ,

0n

n n

x

x x

x

PDP

An algorithm for computing eigenvectors, eigenvalues of positive

definite matrices

• Generally to compute eigenvalues of a matrix we need to first solve the equation for all values of .– |A – I| = 0 (a polynomial of degree n in )

• Then solve the equation for the eigenvector

xxA

, , x

Recall that if A is positive definite then

1 1 1 n n nA x x x x

jixxxx

xxx

jiii

n

if 0 and 1 i.e.1.length of

rseigenvecto orthogonal theare ,,, where 21

It can be shown that

seigenvalue theare 0 and 21 n

222

2211

21

2nnn xxxxxxA

and that 222111 nnmn

mmm xxxxxxA

1111

221

2111 xxxxxxxx m

nn

m

n

m

m

Thus for large values of m

The algorithim

1.Compute powers of A - A2 , A4 , A8 , A16 , ...

2.Rescale (so that largest element is 1 (say))

3.Continue until there is no change, The resulting matrix will be

4.Find

5. Find

constant a 11 xxAm

c 11 xxAm

c that so 11 xxbbAb m

11111 using and 1

xxAbbb

x

To find

6. Repeat steps 1 to 5 with the above matrix to find

7. Continue to find

:Note and 22 x

222111 nnn xxxxxxA

22 and x

nnxxx and ,, and , and 4433

Example

A =5 4 24 10 12 1 2

1 2 3

eigenvalue 12.54461 3.589204 0.866182eignvctr 0.496986 0.677344 0.542412

0.849957 -0.50594 -0.146980.174869 0.534074 -0.82716

Differentiation with respect to a vector, matrix

1

p

df x

dxdf x

dxdf x

dx

Differentiation with respect to a vector

Let denote a p × 1 vector. Let denote a function of the components of .

x f x

x

1 1

then

p

p

f x

x adf x

adx

af x

x

1. Suppose 1 1 n nf x a x a x a x

Rules

1

then 2

p

f x

xdf x

Axdx

f x

x

2. Suppose

2 211 1 pp pf x x Ax a x a x

12 1 2 13 1 3 1, 12 2 2 p p p pa x x a x x a x x

1 1i.e. 2 2 2i ii i ip p

i

f xa x a x a x

x

1122 0 or

df xAx b x A b

dx

Example

f x x Ax b x c

1. Determine when

is a maximum or minimum.

solution

2 2 0

dg xAx x

dx

f x x Ax

2. Determine when is a maximum if1.x x

let 1g x x Ax x x

is the Lagrange multiplier.

solution

or Ax x

Assume A is a positive definite matrix.

This shows that is an eigenvector of A. x

and f x x Ax x x

Thus is the eigenvector of A associated with the largest eigenvalue, .

x

11 1

1

p

ij

q pp

f X f X

x xdf X f X

dX xf X f X

x x

Differentiation with respect to a matrix

Let X denote a q × p matrix. Let f (X) denote a function of the components of X then:

1lnthen

d XX

dX

Example

Let X denote a p × p matrix. Let f (X) = ln |X|

Solution

1 1i i ij ij ip ipX x X x X x X

= (i,j)th element of X-1ln 1

ijij

XX

x X

Note Xij are cofactors

trthen

d AXA

dX

Example

Let X and A denote p × p matrices.

Solution

1 1

trp p

ik kik k

AX a x

trji

ij

AXa

x

Let f (X) = tr (AX)

111

1

p

ij

q qp

dudu

dx dxdudU

dx dxdu du

dx dx

Differentiation of a matrix of functions

Let U = (uij) denote a q × p matrix of functions of x then:

1.

d aU dUa

dx dx

Rules:

2.

d U V dU dV

dx dx dx

3.

d UV dU dVV U

dx dx dx

1

1 14. d U dU

U Udx dx

1U U I 1

1 0p p

dU dUU U

dx dx

Proof:

11dU dU

U Udx dx

11 1dU dU

U Udx dx

tr5. tr

d AU dUA

dx dx

1 1

trp p

ik kii k

AU a u

Proof:

1 1

trtr

p pki

iki k

AU u dUa A

x x dx

11 1tr

6. trd AU dU

AU Udx dx

Proof:

1

1 1tr7. tr ij

ij

d AXE X AX

dx

1

1 1 1 1trtr tr ij

ij ij

d AX dXAX X AX E X

dx dx

1 ,( ) where

0 otherwisekl kl kl

ij ij

i k j lE e e

1 1tr ijE X AX

11 1tr

8. d AX

X AXdX

The Generalized Inverse of a matrix

Recall

B (denoted by A-1) is called the inverse of A if

AB = BA = I

• A-1 does not exist for all matrices A

• A-1 exists only if A is a square matrix and |A| ≠ 0

• If A-1 exists then the system of linear equations has a unique solutionAx b

1x A b

Definition

B (denoted by A-) is called the generalized inverse (Moore – Penrose inverse) of A if

1. ABA = A

2. BAB = B

3. (AB)' = AB

4. (BA)' = BA

Note: A- is unique

Proof: Let B1 and B2 satisfying

1. ABiA = A

2. BiABi = Bi

3. (ABi)' = ABi

4. (BiA)' = BiA

Hence

B1 = B1AB1 = B1AB2AB1 = B1 (AB2)'(AB1) '

= B1B2'A'B1

'A'= B1B2'A' = B1AB2 = B1AB2AB2

= (B1A)(B2A)B2 = (B1A)'(B2A)'B2 = A'B1'A'B2

'B2

= A'B2'B2= (B2A)'B2

= B2AB2 = B2

The general solution of a system of Equations

Ax b

x A b I A A z

The general solution

x A b I A A z

where is arbitrary

Suppose a solution exists

0Ax b

x A b I A A z

Let

then Ax A A b I A A z

AA b A AA A z

0 0AA Ax Ax b

Calculation of the Moore-Penrose g-inverse

1then A A A A

1 1

A A A A A A A A A A I

Let A be a p×q matrix of rank q < p,

Proof

and AA A AI A A AA IA A thus

also is symmetricA A I

1

and is symmetricAA A A A A

1then B B BB

1 1

BB B B BB BB BB I

Let B be a p×q matrix of rank p < q,

Proof

and BB B IB B B BB B I B thus

also is symmetricBB I

1

and is symmetricB B B BB B

1 1then C B BB A A A

1 1 1

CC AB B BB A A A A A A A

Let C be a p×q matrix of rank k < min(p,q),

Proof

is symmetric, as well as

then C = AB where A is a p×k matrix of rank k and B is a k×q matrix of rank k

1 1 1

C C B BB A A A AB B BB B

1

Also CC C A A A A AB AB C

1 1 1

and C CC B BB B B BB A A A

1 1

B BB A A A C