1 Multivariate Linear Regression Models Shyh-Kang Jeng Department of Electrical Engineering/...

11

Multivariate Linear Multivariate Linear Regression ModelsRegression Models

Shyh-Kang JengShyh-Kang JengDepartment of Electrical Engineering/Department of Electrical Engineering/Graduate Institute of Communication/Graduate Institute of Communication/

Graduate Institute of Networking and MultiGraduate Institute of Networking and Multimediamedia

22

Regression AnalysisRegression Analysis

A statistical methodology A statistical methodology – For predicting value of one or more For predicting value of one or more

response (dependent) variablesresponse (dependent) variables– Predict from a collection of predictor Predict from a collection of predictor

(independent) variable values (independent) variable values

33

Example 7.1 Fitting a Straight LineExample 7.1 Fitting a Straight Line

Observed dataObserved data

Linear regression modelLinear regression model

zz11 00 11 22 33 44

yy 11 44 33 88 99

110)( responseMean zYE

44


0 321 4 5

z

y

2

4

6

8

10

0

55

Classical Linear Regression ModelClassical Linear Regression Model

1,

1

1

1

0

2

1

1

0

21

22221

11211

2

1

110

j

nrnrnn

r

r

n

rr

z

zzz

zzz

zzz

Y

Y

Y

zzY

εβZY

66

Classical Linear Regression ModelClassical Linear Regression Model

Iεεε

ε2

2

)'()Cov(

0)(

,0),Cov(

)Var(

0)(

E

E

kj

E

kj

j

j

77

Example 7.1Example 7.1

43210

11111'

98341'

,,

1

1

1

,

5

2

1

1

0

51

21

11

5

2

1

Z

y

εβZY

εZβY

z

z

z

Y

Y

Y

88

Examples 6.6 & 6.7Examples 6.6 & 6.7

level 1% at the rejected is 0:

27.13)01.0(5.195/10

2/78

/SS

)1/(SS

53-3)2(3 d.f.,10SS

213 d.f. ,78SS

128SS,216SS

011

11

121

222

33

444

444

44

444

213

20

969

3210

5,2

H

Fgn

gF

res

tr

res

tr

meanobs

99

Example 7.2 One-Way ANOVAExample 7.2 One-Way ANOVA

11100000

00011000

00000111

11111111

'

21320969'

otherwise0

population from isn observatio theif1

,,,

,,

3322110

3322110

333222111

Z

Y

jz

zzzY

eXeXeX

j

jjjjj

jjjjjj

1010

Method of Least SquaresMethod of Least Squares

βZyyyyβZyε

bβ

ZbyZby

b

b

b

ˆˆ fitted,ˆˆˆ

ˆˆˆˆ residuals

)(minargˆ

'

)(

minimize toas so Selects

110

1

2110

jrrjjj

n

jjrrjj

zzyε

S

zbzbbyS

1111

Result 7.1Result 7.1

βZyyHIyyyyεε

yZyZεyεZ

yHIyyε

ZZZZHHyβZy

yZZZβZ

ˆ''ˆ'ˆ'ˆ

ˆ'',0ˆ'ˆ,0ˆ'

ˆˆ

')'(,ˆˆ

')'(ˆ,1rank full has 1

1

nr

1212

Proof of Result 7.1Proof of Result 7.1

)(minargˆ

0)(')''('

)'''(''ˆ

ˆ'ˆ2

ˆ''ˆˆ'ˆ

')(

ˆˆˆˆ

''ˆ

1

1

1

bβ

ZZyZZZZZIy

ZZZZZIyZβZy

bβZβZy

bβZZbββZyβZy

ZbyZbyb

bβZβZyZbβZβZyZby

yZZZβ

bS

S

1313


yyyyyZZZZIy

yZZZZZZZZZZZZIy

yZZZZIZZZZIy

yyyyεε

εZβεy

yZZZZIZ

βZyZyyZεZ

ˆ'')''('

)''''''2('

)'')(''('

)ˆ()'ˆ(ˆ'ˆ

0ˆ''ˆˆ'ˆ

0)''('

ˆ')ˆ('ˆ'

1

111

11

1

1414


Observed dataObserved data

Linear regression modelLinear regression model

zz11 00 11 22 33 44

yy 11 44 33 88 99

110)( responseMean zYE

1515


6ˆ'ˆ equations of sum residual

'01210ˆˆ

'97531ˆˆ,21ˆ

2

1''

ˆ

ˆˆ,

70

25'

1.02.0

2.06.0',

3010

105'

1

1

0

1

εε

yyε

βZy

yZZZβyZ

ZZZZ

zy

1616

Coefficient of DeterminationCoefficient of Determination

n

jj

n

jj

n

jj

n

jj

n

jj

n

jj

n

jj

n

jj

n

jj

n

jj

yy

yy

yyR

yyyy

ynyn

yyyy

1

2

1

2

1

2

1

2

2

1

2

1

2

1

2

22

111

ˆˆ

1

ˆˆ

ˆ'ˆˆˆ'ˆ'

ˆˆˆˆ'00ˆ'

ˆ'ˆˆ'ˆ)ˆˆ()'ˆˆ()ˆˆ()'ˆˆ('

0ˆ'ˆ

εεyyyy

ε1εZ

εεyyεyεyyyyyyyyy

εy

1717

Geometry of Least SquaresGeometry of Least Squares

plane modelˆ,plane modelon ˆˆ),(minargˆ

)()'()(

plane) modelin (vector - vector)observed(

1

1

1

)( 2

1

1

21

11

10

εβZybβ

ZbyZbyb

Zby

εZβY

ZβY

bS

S

z

z

z

z

z

z

E

nr

r

r

r

n

1818

Geometry of Least SquaresGeometry of Least Squares

1919

Projection MatrixProjection Matrix

βZyZZZZyqqyqq

qqq

y

qqZeZeZZZZ

eeZeZeqqZeq

eeeeeeZZ

eeeeeeZZ

ˆ''

is ,,,

by dconstructe plane model on the of projection

'''

'

111'

'

11

1

'1

1

'

121

1

1

'1

1

'11

'2/12/1'2/12/1'2/1

'

1

'22

2

'11

1

1

'1

'222

'111

r

iii

r

iii

r

r

iii

r

iiii

ikkkikikikikiiii

rrr

rrr

2020

Result 7.2 Result 7.2

eduncorrelat are ˆ and ˆ

)(,1

)('

)1(

ˆ'ˆ

1ˆ'ˆ

''ˆCov,0ˆ

)(ˆˆ

')ˆCov(,ˆ

''ˆ,

222

2

212

12

1

εβ

YHIYεε

εε

HIZZZZIεε

YHIβZYε

ZZβββ

YZZZβεZβY

sErnrn

s

rnE

E

E

2121


''

''')Cov('')ˆCov(

0)('')ˆ(

'')Cov('')ˆ(Cov

)('')ˆ(

''

''''ˆ

'')(''''ˆ

12

11

1

1211

1

1

11

111

ZZZZI

ZZZZIεZZZZIε

εZZZZIε

ZZZZZεZZZβ

βεZZZββ

εZZZZI

εZβZZZZIYZZZZIε

εZZZβεZβZZZYZZZβ

εZβY

EE

EE

2222


)1(

''tr''tr

)'(''tr'ˆˆ

'''tr

'''tr'''

''''''ˆˆ

0''''

'')'(''ˆˆ)ˆ,ˆCov(

2

12212

1

1

11

11

112

11

rn

tr

EE

EE

ZZZZIZZZZI

εεZZZZIεε

εεZZZZI

εZZZZIεεZZZZIε

εZZZZIZZZZIεεε

ZZZZIZZZ

ZZZZIεεZZZεββεβ

2323

Result 7.3Result 7.3Gauss Least Square TheoremGauss Least Square Theorem

βc

Ya'

βc

βcβcβcβc

IεεεZβY

'for unbiased arethat

form theofestimator all among

variancepossiblesmallest thehas ' of

estimatoran as ˆˆˆˆ'

)Cov(,0)(,

2211

1100

2

nn

rr

YaYaYa

E

2424


(BLUE) ˆ'*'' when minimum is

**'*'*

**'**

')'Var(''Var'Var

'*',*''''ˆ'

')ˆ'(

'''')''()'(

,' ofestimator unbiasedan as 'For

2

2

2

1

βcYaYa

aaaaaa

aaaaaa

aIaεaεaZβaYa

cZaYaYZZZcβc

βcβc

cZaβcZβaεaZβaYa

βcYa

E

EE

2525


21

22

22

121

1

2

:ˆ'ˆˆ

ofestimator likelihood maximum:ˆ

ˆ oft independen

',:ˆ

''ˆestimator squaresleast the

as same theis ofestimator likelihood maximul

),(:,

rn

r

n

n

N

N

εε

ZβYε

ZZββ

YZZZβ

β

I0εεZβY

2626


oft independen,''ˆ

)()'(minarg,maxarg

, fixedFor

2

1

2

1

2

1,

1

2

2

2/)()'(2/

2/'2/

2/

1

2

2

222

yZZZβ

ZβyZβyβ

β

ββ

ZβyZβy

εε

L

e

eeL

nn

nn

n

j

j

2727


n

n

L

εε

βZyβZy

β

ˆ'ˆ

'ˆ'ˆ

),ˆ(maxargˆ 22

2

2828


SxxxxΣ

ΣΣμ

xμ

μxΣμxxxxxΣ

Σμ

xxxxΣ

n

n

n

eL

n

L

n

jjj

nnp

n

jjj

n

jjj

1'

1ˆ

2

1),ˆ(

ˆ

'2

1'tr

2

1

:),( ofExponent

1

'tr

2/2/

1

1

1

1

1

2929


''|'

|'

')Cov(

ˆ

Cov

''

''

ˆ

1

1

2

1

1

ZZZZI0

0ZZ

AεA

ε

β

Aεαε

ZZZZI

ZZZ

0

β

ε

β

3030


'11

'22

'11

1

1121

211

1

2

121

1

''

0,1

''tr

1''tr

1,0

''''

''

rnrn

nrnrnrn

n

rn

eeeeeeZZZZI

ZZZZI

ZZZZI

ee

ZZZZIZZZZI

eeZZZZI

3131


2

122

12

22

1

1

1

'12

2

2''

1

'1

'2

'1

1

2

1

:

''''ˆ'ˆˆ

),0(t independen :

)Cov(),Cov(

)Cov(,:

rnrn

rn

iii

i

ikkiki

rn

rnrn

VVV

n

NV

VV

N

V

V

V

εeeεεZZZZIεεε

eεe

V0ε

e

e

e

V

3232

22 Distribution Distribution

)2,12/on with distributi (Gamma

0,0

0,)2/(2

1)(

(d.f.) freedom of degrees :,

)1,0(:);,(:

,),,(:),,(:

2

22/12/22/2

2

1

2

2

2222

2111

2

n

enf

x

NX

ZNX

NXNX

n

nn

i i

ii

i

iii

3333


i

i

rnrii

rnr

n

s

Fr

Fsr

N

ˆ toingcorrespond

' ofelement diagonal:)ˆ(Var

)()1()ˆ(Varˆ

for intervals confidence )%-(1100 usSimultaneo

)1(ˆ''ˆ

for region confidence )%-(1100

),0(:,

12^

1,1

^

i

1,12

2

ZZ

ββZZββ

β

IεεZβY

3434


)()1('ˆ''ˆ

region Confidence

:)1/(1

)1/('

:1

:'),,0(:

')ˆCov(')Cov(

0)(,ˆ'

1,12

1,12

21

22

21

221

22/12/1

2/1

rnr

rnr

rn

rr

Fsr--

Frnsrn

r

srn

N

E

VVββZZββ

VV

VVIV

IZZβZZV

VββZZV

3535

Example 7.4 (Real Estate Data)Example 7.4 (Real Estate Data)20 homes in a Milwaukee, Wisconsin, 20 homes in a Milwaukee, Wisconsin, neighborhoodneighborhood

Regression modelRegression model

dollars) of (thousands valueassessed:

feet) squared of (hundereds size dwelling total:

dollars) of (thousands price selling:

2

1

22110

z

z

Y

zzY jjj

3636


647.0,556.0:for interval confidence %95

285.0110.2045.0)ˆ(Var025.0ˆ

834.0,045.0634.2967.30ˆ

285.00067.0,785.00512.0,88.71523.5

473.3,

045.0

634.2

967.30

''ˆ

0067.00172.01463.0

0512.02544.0

1523.5

'

2

2

^

172

22

)285.0(1

)785.0()88.7(

1

1

t

Rzzy

sss

syZZZβ

ZZ

3737


yZZZβ

βZyβZyβZyβZy

ZZ

εβZβZεβ

βZZεZβY

I0εβ

ZZ

β

''ˆ

)ˆ()'ˆ()ˆ()'ˆ(

)(SS)(SS

),(:,

)()/()(SS)(SS

if 0: rejects test ratio Likelihood

11

11)1(

)1(1)1(1

1

)2(2)1(1)2(

)1(21

221

')2(

1,21

)2(0

resres

nrqq

rnqrresres

N

Fs

qr

H

3838

Effect of RankEffect of Rank

In situations where In situations where ZZ is not of full rank, is not of full rank, rrank(ank(ZZ)) replaces replaces r+1r+1 and and rank(rank(ZZ11)) replaces replaces q+1q+1 in Result 7.6 in Result 7.6

3939


)ˆ()'ˆ(ˆ and ''ˆat occurs

maximum the where,ˆ2

1),(max

and ,0:Under

)ˆ()'ˆ(ˆ and ''ˆat occurswhich

ˆ2

1

2

1max),(max

)1(1)1(1211

111(1)

2/

12/

2)1(

,

)1(1)2(0

21

2/2/

2/)()'(2/

,

2

,

2(1)

2

22

βZyβZyyZZZβ

β

εβZYβ

βZyβZyyZZZβ

β

β

ZβyZβy

ββ

n

nn

n

nn

nn

eL

H

e

eL

4040


1,2

122

12

122

21

2

221

22210

2/

2

221

2/

2

21

2

,

2)1(

,

)2(0

:,:ˆ,:ˆ

)/()(SS)(SS

)1/(ˆ)/(ˆˆ

or ˆ/ˆˆ largefor reject toequivalent is

ˆ

ˆˆ1

ˆ

ˆ

),(max

),(max

of valuessmallfor 0:Reject

2

2(1)

rnqrqnrn

resres

nn

FFnn

Fs

qr

rnn

qrn

H

L

L

H

ZZ

β

β

β

β

β

4141

Wishart DistributionWishart Distribution

)'|'(:')|(:

)|(:

)|(:),|(:

:Properties

definite positive:

21

2

)|(

2121

2211

1

2/)1(4/)1(2/)1(

2/tr2/)2(

1

21

21

1

CΣΣCACCACΣAA

ΣAAAA

ΣAAΣAA

A

Σ

AΣA

AΣ

mm

mm

mm

p

i

nppnp

pn

n

WW

W

WW

in

ew

4242

Generalization of Result 7.6Generalization of Result 7.6

2211

12

1,2

11

0

:ˆ)''('ˆ

and )')'(,(:ˆ since

)( ˆ)''('ˆ

if levelat :Reject

matrix )1()(:

rn

qr

rnqr

NC

Fqrs

H

rqr

βCCZZCβC

CZZCCββ

βCCZZCβC

0Cβ

C

4343

Example 7.5 (Service Ratings Example 7.5 (Service Ratings Data)Data)

4444

Example 7.5: Design MatrixExample 7.5: Design Matrix

4545

Example 7.5 Example 7.5

tsignifican isgender hebut that t

effect,location no is thererify that further vecan We

level eappropriatany for ant insignific :89.0

12/)(SS

2/)(SS)(SS)46/()(SS)(SS

0:

14418)rank(,1.3419)(SS

of columnssix first :

12)rank(,4.2977)(SS,6)rank(

'

12

1

3231222112110

11

1

323122211211213210

Z

ZZZZ

ZZ

ZZ

ZZZ

β

res

resresresres

res

res

sF

H

n

n

4646


20

1'01

'0

00

20

1'0

'0

00'0

'00011000

00001'0

2

'2/ˆ

is | of interval congidence )%1(100

'ˆVar

variance.minimumwith

| ofestimator unbiased theis ˆ

|

at response:,1

),0(:,

st

YE

YE

zzYE

Yzz

N

rn

rr

r

n

zZZzβz

z

zZZzβz

zβz

βzz

zz

IεεZβY

4747


1

01'

02

'0

'0

22

01'

02'

0'0

20

1'0

'0

'0

21

22

121

20

1'00

'00

'0

'0

i'0

:'

ˆ

/

'/ˆ

)',(:ˆ

)1/(:/ oft independen iswhich

)',(:ˆ

')ˆCov(ˆVar

7.3Result by varianceminimum

with the ofestimator unbiased theis ˆ

s' ofn combinatiolinear a is

rn

rn

r

tss

N

rns

N

zZZz

βzβzzZZzβzβz

zZZzβzβz

ZZββ

zZZzzβzβz

βzβz

βz

4848


01'

02

1'0

0

01'

02'

00

'00

220

0'00

'12/ˆ

:for interval prediction )%1(100

'1)ˆ Var(

ˆ: ofpredictor unbiased

,ˆ , oft independen ),0(:

zZZzβz

zZZzβz

βz

βε

βz

st

Y

Y

Y

sN

Y

rn

4949


1

01'

02

'00

21

22

01'

02'

00

01'

02

'00

'00

'00

'00

'00

'00

'0

'00

:'1

)ˆ(

:)1(

)'1,0(:)ˆ(

'1

))ˆ(Var()Var()ˆVar(

0))ˆ(()()ˆ(

)ˆ(ˆˆerror Forecast

rn

rn

ts

Y

srn

NY

Y

EEYE

Y

zZZz

βz

zZZzβz

zZZz

ββzβz

ββzβz

ββzβzβzβz

5050

Example 7.6 (Computer Data)Example 7.6 (Computer Data)

5151


155.86) (148.08,or '1)025.0(ˆ

at interval prediction %95

153.94) (150.00,or ')025.0(ˆ

timeCPUmean for the interval confidence %95

776.2)025.0(,71.0'

97.1515.7*42.0130*08.142.8ˆ,204.1

01440.000107.008831.0

00052.006411.0

17969.8

'

42.008.142.8ˆ,5.71301

01'

04'0

0

01'

04'0

401'

0

'0

1

21'0

zZZzβz

z

zZZzβz

zZZz

βz

ZZ

z

st

st

ts

s

zzy

5252

Adequacy of the ModelAdequacy of the Model

),0(: of estimatean isˆ

')'(ˆ

ˆˆˆˆ

ˆˆˆˆ

ˆˆˆˆ

2

1

110

2211022

1111011

N

zzy

zzy

zzy

jj

nrrnnn

rr

rr

yHIyZZZZIε

5353

Residual PlotsResidual Plots

5454

Q-QQ-Q Plots and Histograms Plots and Histograms

Used to detect the presence of Used to detect the presence of unusual observations or severe unusual observations or severe departures from normality that may departures from normality that may require special attention in the require special attention in the analysisanalysis

If If nn is large, minor departures from is large, minor departures from normality will not greatly affect normality will not greatly affect inferences about inferences about

5555

Test of Independence of TimeTest of Independence of Time

)1(2

ˆ

ˆˆ

TestWatson -Durbin

ˆ

ˆˆ

ationautocorrelfirst thefrom dconstructeTest

1

1

2

2

21

1

2

21

1

r

r

n

jj

n

jjj

n

jj

n

jjj

5656

Example 7.7: Residual PlotExample 7.7: Residual Plot

5757

LeverageLeverage

““Outliers” in either the response or Outliers” in either the response or explanatory variables may have a explanatory variables may have a considerable effect on the analysis considerable effect on the analysis and determine the fitand determine the fit

Leverage for simple linear Leverage for simple linear regression with one explanatory regression with one explanatory variable variable zz

n

r

zz

zz

nh n

jj

jjj

1 average,

1

1

2

2

5858

Mallow’s Mallow’s CCpp Statistic Statistic

Select variables from all possible Select variables from all possible combinationscombinations

)2(model fullfor varianceresidual

interceptan including ,parameters with

modelssubset for squares of sum residual

pnp

C p

5959

Usage of Mallow’s Usage of Mallow’s CCpp Statistic Statistic

6060

Stepwise RegressionStepwise Regression1. The predictor variable that 1. The predictor variable that explains the largest significant explains the largest significant proportion of the variation in proportion of the variation in YY is the is the first variable to enterfirst variable to enter2. The next to enter is the one that 2. The next to enter is the one that makes the highest contribution to makes the highest contribution to the regression sum of squares. Use the regression sum of squares. Use Result 7.6 to determine the Result 7.6 to determine the significance (significance (FF-test) -test)

6161

Stepwise RegressionStepwise Regression3. Once a new variable is included, 3. Once a new variable is included, the individual contributions to the the individual contributions to the regression sum of squares of the regression sum of squares of the other variables already in the other variables already in the equation are checked using equation are checked using FF-tests. -tests. If the If the FF-statistic is small, the variable -statistic is small, the variable is deletedis deleted4. Steps 2 and 3 are repeated until 4. Steps 2 and 3 are repeated until all possible additions are non-all possible additions are non-significant and all possible deletions significant and all possible deletions are significantare significant

6262

Treatment of ColinearityTreatment of ColinearityIf If ZZ is not of full rank, is not of full rank, Z’ZZ’Z does not have an inv does not have an inverse erse Colinear ColinearNot likely to have exact colinearityNot likely to have exact colinearityPossible to have a linear combination of coluPossible to have a linear combination of columns of mns of ZZ that are nearly 0 that are nearly 0Can be overcome somewhat byCan be overcome somewhat by– Delete one of a pair of predictor variables that are sDelete one of a pair of predictor variables that are s

trongly correlatedtrongly correlated– Relate the response Relate the response YY to the principal components to the principal components

of the predictor variablesof the predictor variables

6363

Bias Caused by a Bias Caused by a Misspecified ModelMisspecified Model

)1(

)2(211

11)1(

)2(2)1(111

1111

11)1(

11

11)1(

)2(2)1(1)2(

)1(21

21

ofestimator biased

''

'')('')ˆ(

''ˆ

β

βZZZZβ

βZβZZZZYZZZβ

YZZZβ

εβZβZεβ

βZZY

ZZZ

EE

6464

Example 7.3Example 7.3Observed dataObserved data

Regression modelRegression model

2112022

1111011

zY

zY

zz11 00 11 22 33 44

yy11 11 44 33 88 99

yy22 -1-1 -1-1 22 33 22

6565

Multivariate Multiple RegressionMultivariate Multiple Regression

nrnn

r

r

jmjjjjmjjj

m

mrrmmmm

rr

rr

zzz

zzz

zzz

YYY

E

zzY

zzY

zzY

10

22120

11110

21'

21'

21

110

22112022

11111011

,

)Var(,0)(,'

Z

εY

Σεεε

6666


)4()3()2()1(

21

11211

00201

)()2()1(

21

22221

11211

βββββ

YYYY

rmrr

m

m

m

nmnn

m

m

YYY

YYY

YYY

6767


ik

ikkii

n

m

nmnn

m

m

mkiE

Σ

Iεεε

εZβY

ε

ε

ε

εεεε

,,2,1,,,Cov,0)( )()()(

'

'2

'1

)()2()1(

21

22221

11211

6868


)()()()()1()1()()(

)()()1()1()1()1()1()1(

)()2()1(1

)()2()1(1

)()2()1(

)(1

)(

)()()()(

''

''

'

,''ˆ

''ˆˆˆˆ

''ˆ

,,2,1,)Cov(,

mmmmmm

mm

m

mm

ii

iiiiii mi

ZbYZbYZbYZbY

ZbYZbYZbYZbY

ZBYZBY

bbbBYZZZβ

YYYZZZββββ

YZZZβ

IεεZβY

6969


ˆby minimized

also is ' variancedGeneralize

ˆby minimized is 'tr

' minimizes ˆ)()()()()()(

βB

ZBYZBY

βBZBYZBY

ZbYZbYβb

iiiiii

7070


βZZβYYYYYYεε

εεYYεYεYYY

0YZZZZIZβεY

0YZZZZIZεZ

YZZZZIYYε

YZZZZβZY

ˆ''ˆ'ˆ'ˆ'ˆ'ˆ

ˆ'ˆˆ'ˆˆˆ'ˆˆ'

''''ˆˆ'ˆ

'''ˆ'

''ˆˆ:Residuals

''ˆˆ : valuesPredicted

1

1

1

1

7171


2211

)2()1(1

)2()1(

)2(1

)2(

)1(1

)1(

1

1ˆ,21ˆ

''12

11ˆˆˆ

'11''ˆ

'21''ˆ

1.02.0

2.06.0',

43210

11111'

zyzy

yyZZZβββ

yZZZβ

yZZZβ

ZZZ

7272


verifiedis ˆ'ˆˆ'ˆ'

42

26ˆ'ˆ,

1545

45165ˆ'ˆ,1943

43171'

00

00ˆ'ˆ'11110

01210ˆˆ

39

27

15

03

11

12

11

41

31

21

11

01

ˆˆ

εεYYYY

εεYYYY

YεYYε

βZY

7373


eduncorrelat are ˆ and ˆ

1

ˆ'ˆ,0ˆ

)1(ˆˆ,ˆ

ˆˆˆˆˆ

'ˆ,ˆCov

ˆor ˆ

)('

)()(

)()2()1(

1)()(

)()(

βε

Σεε

ε

εε0ε

βZYεεεε

ZZββ

ββββ

rnEE

rnEE

EE

ikkii

m

ikki

ii

7474


11'

)()(1

)()()()()()(

)()()(

)(1

)(1

)()()(

)(1

)()(1

)()(

')()()()()()(

''''

'ˆˆˆ,ˆCov

ˆ,ˆ

''

''ˆˆ

''''ˆ

,0,

ZZZZZεεZZZ

ββββββ

0εββ

εZZZZI

YZZZZIYYε

εZZZβYZZZββ

IεεεεZβY

ikki

kkiiki

iii

i

iiii

iiiii

iiiiiiii

E

E

EE

EE

7575


Σεε

ZZZZI

IZZZZI

εZZZZIεεε

UUAAUUAUU

AU

1

ˆˆ

)1(')'(tr

')'(tr

')'(ˆˆ

)'(tr'tr'

matrix fixed: vector,random :

)('

)(

1

1

)(1'

)()('

)(

rnE

rn

EE

EEE

ki

ikik

ik

kiki

7676


0ZZZZZZ

ZZZZIIZZZ

ZZZZIεεZZZ

ZZZZIεεZZZ

εβ

''''

''''

''''

''''

ˆ,ˆCov

11

11

1')()(

1

1')()(

1

)()(

ik

ik

ki

ki

ki

E

E

7777

Forecast ErrorForecast Error

0)ˆ(

ˆˆˆ

,0)(

oft independen

'

'ˆˆ'ˆˆ

)ˆ(,ˆˆ

,1

)('00

)()('00)(

'0)(

'0)(

'00)(

'00

000

00201'0

01'

0

0)()()()('00)()()()(

'0

'0

'0)(

'0)(

'0)(

'0

0)('00001

'0

ii

iiiiiiiii

ikkii

m

ik

kkiikkii

iii

iiir

YE

YY

EE

EE

EEE

Yzz

βz

ββzβzβzβzβz

εε

zZZz

zββββzzββββz

βzβzβzβzβz

βzz

7878

Forecast ErrorForecast Error

0)ˆ(,'ˆ

'1

'ˆ)ˆ(

'ˆˆ

ˆˆ

ˆˆ

0)()()(1

)()(

01'

0

0)()(00)()('0

0)()()()('000

)()('00)()(

'00

)('00)(

'00

kiiiii

ik

kkikii

kkiiki

kkkiii

kkii

E

EE

EE

E

YYE

ββεZ'ZZββ

zZZz

zββββz

zββββz

ββzββz

βzβz

7979


)( as ddistribute is ˆ and

ˆ'ˆ1ˆ'ˆ

1ˆby given

ofestimator likelihood maximum theoft independen is ˆ

'ˆ,ˆCov

ˆon with distributi normal:ˆ

ofestimator likelihood maximum theis ˆ

ondistributi normal:,

1,

1)()(

ΣΣ

βZYβZYεεΣ

Σβ

ZZββ

βββ

ββ

εεZβY

rnp

ikki

Wn

nn

E

8080


2)(

1

11

0

)1(1)1(11,1

1,

)2(0

))(()2(

))1(()1(

~ˆ

ˆln2/)1(1 large, For

ˆˆˆ

ˆln

ˆ

ˆlnln

of valueslargefor Reject

ˆ'ˆˆ,:ˆˆ

oft independen :ˆ'ˆˆ

:,

qrm

qrp

rnp

mqr

mq

qrmrnn

nn

nnnn

H

nWn

Wn

H

Σ

Σ

ΣΣΣ

Σ

Σ

Σ

βZYβZYΣΣΣΣ

ΣβZYβZYΣ

0β

β

β

β

8181


0

2)(

1

111

111

)2(0

)2(

1

reject not do ,49.9)05.0(

28.3ˆˆˆ

ˆln)1(1

31)rank(,51)rank(

,05.0,:

parametersn interactiogender locatin:

12.36616.246

16.24676.441ˆˆ

95.205072.1021

72.102139.2977ˆ

index.quality -service more one plus data 7.5 Example

11H

nn

nqrmrn

qr

H

n

n

qrm

ΣΣΣ

Σ

ZZ

0β

β

ΣΣ

Σ

8282

Other Multivariate Test StatisticsOther Multivariate Test Statistics

1

1

1

1

1

1

1

121

1

1 root greatest sRoy'

trceLawlay tra-Hotelling

1

1)(tr tracesPillai'

1

1 lambda sWilk'

,min , of seigenvalue:

ˆˆ,ˆ

s

ii

s

i i

s

i i

s qrps

nn

HE

EHH

HE

E

HE

ΣΣHΣE

8383

Predictions from RegressionsPredictions from Regressions

)()1(

'

''ˆ1

ˆ'''ˆ

:'for ellipsoid confidence )%-100(1

'

''ˆ

1

ˆ'

'

''ˆ

as ddistributetly independen:ˆ

',':'ˆ7.10Result

,01'

0

00

1

00

0

01'

0

00

1

01'

0

002

1

01'

000

mrnm

rn

m

Fmrn

rnm

rn

n

rn

nT

Wn

N

zZZz

zβzβΣ

zβzβ

zβ

zZZz

zβzβΣ

zZZz

zβzβ

ΣΣ

ΣzZZzzβzβ

8484


Σ

zZZz

βz

βz

ˆ ofelement diagonalth the:ˆ

ˆ1

')()1(

ˆ

:)(for

intervals confidence ussimultaneo )%1(100

01'

0,

)('0

)('0

i

rn

nF

mrn

rnm

YE

ii

iimrnm

i

ii

8585


iimrnmi

i

mrnm

m

rn

nF

mrn

rnm

Y

Fmrn

rnm

rn

n

n

N

ˆ1

)'1()()1(ˆ

:for intervals prediction ussimultaneo )%1(100

)()1(

)'1(

'ˆˆ1

''ˆ

:for ellipsoid prediction )%1(100

ˆ oftly independen

))'1(,0(:'ˆ'ˆ

,'

01'

0,)('0

0

,01'

0

00

1

00

0

01'

00000

000

zZZzβz

zZZz

zβYΣzβY

Y

Σ

ΣzZZzεzββzβY

εzβY

8686


13.1330.5

30.580.5

ˆ'ˆˆ'ˆ

ˆ'ˆˆ'ˆˆ

17.349ˆ,97.151ˆ

4208.142.8ˆ 7.6, Example From

34725.0','67.525.214,14ˆ

812.1,67.525.214.14ˆ :equation Fitted

capacity I/Odisk : data 7.6 Example

)2()2()2()2()1()1()2()2(

)2()2()1()1()1()1()1()1(

)2('0)1(

'0

)1(

0'0)2(

212

2

βZyβZyβZyβZy

βZyβZyβZyβZyΣ

βzβz

β

zZZzβ

n

szzy

Y

8787


34725.1'1

with34725.0' replace :ellipse prediction 95%

)05.0(3

4234725.0

17.349

97.151

13.1330.5

30.580.517.349,97.151

:'for ellipse confidence 95%

2,2,7

17.349

97.151ˆ

ˆ

ˆ

ˆ'ˆ

01'

0

01'

0

3,2

)2('0

)1('0

1

)2('0)1(

'0

0

)2('0

)1('0

0')2(

')1(

0

zZZz

zZZz

βz

βzβzβz

zβ

βz

βzz

β

βzβ

F

mrn

8888


8989

Linear RegressionLinear Regression

20

0

022110

'

21

21

' error squaremean

' error prediction

' predictor linear

,

:matrix covariance and mean

normaly necessarilnot :),,,,(

variablesrandom:,,,,

Zb

Zb

Zb

Σσ

σΣ

μμ

Σμ

ZZZ

Z

bYE

bY

bZbZbZbb

zzzyf

ZZZY

rr

Y

YYY

Z

Y

r

r

9090


YY

YY

YY

b

YYYY

YY

bYY

Y

YE

Y

ZZZZZZ

b

ZZZZ

ZZZZ

σΣσβΣβ

ZbZβ

Zβ

σΣσZβ

μβσΣ

Zβ

1'

0,

0

0

1'20

01

0

0

'

',Corrmax',Corr

n with correlatio

maximum havingpredictor linear theis ' Also,

)'(

',

response theof predictorslinear all

among squaremean minimum has 'predictor Linear

0

9191


ZZZZZZZZ

ZZZZ

ZZZZZZZZZ

ZZZZ

Z

ZZ

ZZ

ZZ

μσΣμbβσΣb

σΣσ

μbσΣbΣσΣb

σbμbbΣb

μZb

μbμZb

μbμZb

Zb

μbμbZbZb

'',at minimized

''

'2''

'2

''

''

'

''''

10

1

1'

20

11

20

20

22

20

20

00

YYYY

YY

YYYYY

YYYY

Y

YY

YY

YY

b

b

b

YE

bEYE

bYE

bYE

bb

9292


βΣββΣΣσβσσΣσ

βσΣb

σΣσZb

σΣbσΣbσb

bΣb

σbZb

bZb

ZZZZZZZZZZZZ

ZZZ

ZZZZ

ZZZZZZZ

ZZ

Z

Z

'

for equality with

,'Corr

''

inequality Schwartz-Cauchy Extended

'

','Corr

','Cov

1''1'

1

1'2

0

1'2

2

20

0

YYYY

Y

YY

YY

YYY

YY

Y

Y

Yb

Yb

Yb

9393

Population Multiple Population Multiple Correlation CoefficientCorrelation Coefficient

error no with predicted becan :1

in power predictive no:0

1

:forecast to' usingin error squareMean

:iondeterminat oft coefficien Population

:tcoefficienn correlatio multiple Population

2)(

2)(

2)(

1'

0

2)(

1'

)(

Y

Yβ

Y

Y

YYYYYYY

Y

YY

YYY

Z

Z

ZZZZZ

Z

ZZZZZ

Z

σΣσ

Zβ

σΣσ

9494


548.010

3

7 error squaremean

23' :predictorlinear best

3','21

23|1

37|1

11|10

|

|

,

0

2

5

1'

)(

1'

210

01

'

YY

YYY

YYYY

YY

Y

YYYY

ZZ

ZZZZZ

ZZZZ

ZZZZ

ZZZ

Z

Z

σΣσ

σΣσ

Zβ

μβσΣβ

Σσ

σ

Σ

μ

μ

9595

Linear Predictors and NormalityLinear Predictors and Normality

error. squaremean smallest with the

predicts stillit ss,Neverthele linear. benot need

| normal,not is population When the

function) regression(linear

')(|

),(

: with ofon distributi lConditiona

),(:'

01'

1'1'

121

Y

YE

YE

N

Y

NZZZY

YY

YYYYYY

rr

z

zβμzΣσz

σΣσμzΣσ

zZ

Σμ

zZZZ

ZZZZzZZZ

9696


YYYYYY

Y

YY

Y

YYY

r

sn

n

YE

Y

YY

sY

NY

ZZZZZ

ZZZ

ZZZZZZ

ZZZ

Z

sSs

Zβ

ZzSszβ

ZβZSssSβ

Ss

sS

Zμ

ΣμZ

1'

20

1'0

1'0

1

'

1

1ˆ

' ofestimator likelihood maximum

)('ˆˆ

estimator likelihood maximum

'ˆˆ,ˆ

tscoefficien theofestimator likelihood maximum

,ˆ

),(: and ofon distributiJoint

9797


1

'ˆˆ

1

1

:for estimator Unbiased

and for 1ˆ and ˆ estimators

likelihood maximum theofon substitutiby conclusion get the to

error squaremean

'

,,'

and ,likelihood maximum ofproperty invariance Apply the

1

2

01'

1'

1'0

110

rn

Y

srn

n

n

n

n

jjj

YYYY

YY

YYYYYY

YY

YYY

Zβ

sSs

ΣμSΣμ

σΣσ

μzΣσzβ

σΣβμσΣ

ZZZZ

Z

ZZZZZ

ZZZZ

ZZZZZZZ

9898

Invariance PropertyInvariance Property

)Var( of MLE1

ˆ

ˆ of MLE

ˆˆ'ˆ' of MLE

:Examples

)( ofestimator likelihood maximum:)ˆ(

ofestimator likelihood maximum:ˆ

1

2

11

i

n

jijiii

iiii

XXXn

hh

μΣμμΣμ

9999


657.13034.28|983.35

034.28200.377|763.418

983.35763.418|913.467

|

|

,

547.3

24.130

44.150

ˆ

,,on nsobservatio 7 data, 7.6 Example

'

21

ZZZ

Z

Ss

s

S

z

μ

Y

YYYs

y

ZZYn

100100


894.01

error squaremean theof estimate likelihood maximum

42.008.142.8'ˆˆ

function regression estimated

421.8'ˆˆ,420.0

079.1ˆ

1'

210

01

YYYY

Y

sn

n-

zz

y

ZZZZ

ZZZ

sSs

zβ

zβsSβ

101101

Prediction of Several VariablesPrediction of Several Variables

tscoefficien regression ofmatrix :

of prediction for theerror squaremean theminimizes

|

example,For s.regression univariate of composed

|

: and of regression temultivaria

|

|

,),,(:

1

1

)()(

)()(

)1(

)1(

)1(

)1(

11

1ZZYZ

Z1

ZZZ

Z1

ZZYZY

ZZZY

YZYY

Z

Y

ΣΣβ

μzΣΣz

μzΣΣμzY

ZY

ΣΣ

ΣΣ

Σ

μ

μ

μΣμ

Z

Y

Y

YE

m

E

N

YY

rrmr

rmmm

r

m

rm

r

m

102102


ZY

1ZZYZYYZYY

1ZZYZ

ZY1

ZZYZYYZYY

Z1

ZZYZ

SSSSΣ

ZzSSYzββ

ΣΣΣΣΣβZβYβZβY

μzΣΣβzβZY

ΣμZY

n

n

E

N

Y

rm

1ˆ

ˆˆ

estimators likelihood maximum

'

: and of regression

),(: and

0

00

0

103103


657.13034.28|558.140983.35

034.28200.377|976.1008763.418

558.140976.1008|491.3072556.1148

983.35763.418|556.1148913.467

547.3

24.130

79.327

44.150

ˆ

7.10. and 7.6 Example of Data

S

z

y

μ

104104


205.2893.0

893.0894.01

of estimate likelihood maximum

547.3665.524.130254.2

547.3420.024.130079.1

79.327

44.150

ˆˆ

21

21

0

ZY1

ZZYZYY

ZYY

1ZZYZ

SSSS

Σ

)z(zSSyzββ

n

n

zz

zz

105105

Partial Correlation CoefficientPartial Correlation Coefficient

ZZ

ZZ

ZZ

ZZ

ZY1

ZZYZYYZYY

Z1

ZZZZ1

ZZZ

Z

ΣΣΣΣΣ

μZΣΣμZΣΣ

2211

21

21

2211

21

21

2211

tcoefficienn correlatio partial Sample

: of effects thegeliminatin

, and between t coefficienn correlatio Partial

matrix covarianceError

,

:errors ofPair

21

21

YYYY

YYYY

YYYY

YYYY

YYYY

ss

sr

YY

YY

106106


responsesboth on of effects thegeliminatinafter

reducedsharply been has and between n Correlatio

96.0

64.0572.2043.1

042.1

572.2042.1

042.1043.1

data 7.13 Example

21

21

2211

21

21

Z

SSSS

ZZ

ZZ

ZY1

ZZYZYY

YY

r

ss

sr

YY

YYYY

YYYY

107107

Mean Corrected Form of the Mean Corrected Form of the Regression ModelRegression Model

2'

22'

2'

2

2'

'22

11

2121

1111

111*

110

'''

0,|

1

1

1

matrixdesign correctedmean

)()(

ccccc

ccc

cc

rnrn

rr

rr

c

jrjrrj

jjrrjj

n

zzzz

zzzz

zzzz

zzzz

zzY

ZZ0

0

ZZ1Z

Z111ZZ

1ZZ1Z

108108

Mean Corrected Form of the Mean Corrected Form of the Regression ModelRegression Model

21

2'

2

2

21'

*

**

1

2'

22'

*

'2

1

2'

2'

2

1

2'

2

'1'*

'/

)ˆCov()ˆ,ˆCov(

)ˆ,ˆCov()ˆVar(

'ˆˆˆ

''/1

ˆ

ˆ

cc

cc

cc

c

cccc

cccccc

ccc

c

n

yy

yn

ZZ0

0

ZZββ

β

zzZZZyzzβ

yZZZyZ

y1

ZZ0

0

yZZZβ

109109

Mean Corrected Form for Mean Corrected Form for Multivariate Multiple RegressionsMultivariate Multiple Regressions

ii

ii

ii

zzii

zzii

zziji

iccc

i

sn

sn

snzz

y

i

)1(ˆ~̂

)1(~

)1(/

ablesinput vari edstandardiz

ˆ

:responseth the

for t vectorscoefficien theof estimates squareleast

)('

2

1

2'

2

)(

(i)

yZZZβ

110110

Relating the FormulationsRelating the Formulations

1'1'

1

2'

22

1

2'

22

2222

1'1

2'

22'

0*

1

2'

22'

*

1'0

)1()1(

''

''''

'ˆ'ˆ,ˆˆ

'ˆˆˆ

:form correctedmean

'ˆˆ :7.13Result

ZZZZZZ

ZZZ

ZZZ

SsSs

ZZZ1yZZZy

Z1yZ1Z1yZy

βSsZZZyβ

zzZZZyzzβ

zzSszβ

YY

cccccc

cccc

Ycccc

cccc

Y

nn

y

yyy

y

yy

y

111111


Example 7.6, classical linear Example 7.6, classical linear regression modelregression model

Example 7.12, joint normal Example 7.12, joint normal distribution, best predictor as the distribution, best predictor as the conditional meanconditional mean

Both approaches yielded the same Both approaches yielded the same predictor of predictor of YY11

21 42.008.142.8ˆ zzy

112112

Remarks on Both FormulationRemarks on Both Formulation

Conceptually differentConceptually different

Classical modelClassical model– Input variables are set by experimenterInput variables are set by experimenter– Optimal among linear predictorsOptimal among linear predictors

Conditional mean modelConditional mean model– Predictor values are random variables Predictor values are random variables

observed with the response valuesobserved with the response values– Optimal among all choices of predictorsOptimal among all choices of predictors

113113

Example 7.16 Natural Gas DataExample 7.16 Natural Gas Data

114114

Example 7.16 : First ModelExample 7.16 : First Model

52.0ˆ

ˆˆ

)ˆ( ation autocorrel lag1

But,

intercept except the t,significan are tscoefficien All

952.0

Weekend15.857 - Windspeed1.315

DHDLag 1.405 DHD 5.874 1.858Sendout

1

2

21

1

2

n

jj

n

jjj

r

R

115115

Example 7.16 : Second ModelExample 7.16 : Second Model

negligible all are residuals theof nscorrelatio auto

89.228ˆ

240.0470.0

Weekend10.109 - Windspeed1.207

DHDLag 1.426 DHD 5.810 2.130 Sendout

as model fitted aget toSASApply

noise siveautoregresan with errorst independen theReplace

2

71

7771

jjjj

jjjj

NNN

NNN

1 Multivariate Linear Regression Models Shyh-Kang Jeng Department of Electrical Engineering/...

Documents

Transcript of 1 Multivariate Linear Regression Models Shyh-Kang Jeng Department of Electrical Engineering/...