Multiple Regression

© 2002 Prentice-Hall, Inc. Chap 14-1

Basic Business Statistics(8th Edition)

Chapter 14Introduction to

Multiple Regression


Chapter TopicsChapter Topics

The multiple regression modelThe multiple regression model Residual analysisResidual analysis Influence analysisInfluence analysis Testing for the significance of the Testing for the significance of the

regression modelregression model Inferences on the population Inferences on the population

regression coefficientsregression coefficients Testing portions of the multiple Testing portions of the multiple

regression modelregression model


0 1 1 2 2i i i k ki iY b b X b X b X e

Population Y-intercept

Population slopes Random Error

The Multiple Regression ModelThe Multiple Regression Model

A relationship between A relationship between one dependent one dependent and two or and two or more more independentindependent variables is a variables is a linear functionlinear function

Dependent (Response) variable for sample

Independent (Explanatory) variables for sample model

1 2i i i k ki iY X X X

Residual


Population Multiple Regression Model

X2

Y

X1YX = 0 + 1X 1i + 2X 2i

0

Y i = 0 + 1X 1i + 2X 2i + i

ResponsePlane

(X 1i,X 2i)

(O bserved Y )

i

X2

Y

X1YX = 0 + 1X 1i + 2X 2i

0

Y i = 0 + 1X 1i + 2X 2i + i

ResponsePlane

(X 1i,X 2i)

(O bserved Y )

i

Bivariate model


Sample Multiple Regression Model

X2

Y

X1

b0

Y i = b0 + b1X 1 i + b2X 2 i + e i

ResponsePlane

(X 1i, X 2i)

(O bserved Y)

^

e i

Y i = b0 + b1X 1 i + b2X 2 i

X2

Y

X1

b0

Y i = b0 + b1X 1 i + b2X 2 i + e i

ResponsePlane

(X 1i, X 2i)

(O bserved Y)

^

e i

Y i = b0 + b1X 1 i + b2X 2 i

Bivariate model

Sample Regression PlaneSample Regression Plane


Simple and Multiple Simple and Multiple Regression ComparedRegression Compared

Coefficients in a simple regression pick up the impact of that variable plus the impacts of other variables that are correlated with it and with the dependent variable.

Coefficients in a multiple regression net out the impacts of other variables in the equation.


Simple and Multiple Simple and Multiple Regression Compared: ExampleRegression Compared: Example

Two simple regressionsTwo simple regressions:

Multiple regression:

0 1

0 1

Oil Temp

Oil Insulation

0 1 2Oil Temp Insulation


Multiple Linear Multiple Linear Regression EquationRegression Equation

Too complicated

by hand! Ouch!


Interpretation of Interpretation of Estimated CoefficientsEstimated Coefficients

Slope (Slope (bbii)) Estimated that the average value of Estimated that the average value of YY changes changes

by by bbii for each one unit increase in for each one unit increase in XXii holding all holding all other variables constant (ceterus paribus)other variables constant (ceterus paribus)

Example: if Example: if bb11 = -2, then fuel oil usage ( = -2, then fuel oil usage (YY) is ) is expected to decrease by an estimated two expected to decrease by an estimated two gallons for each one degree increase in gallons for each one degree increase in temperature (temperature (XX11) given the inches of insulation ) given the inches of insulation ((XX22))

Y-intercept (Y-intercept (bb00)) The estimated average value of The estimated average value of YY when all when all XXii = 0 = 0


Multiple Regression ModelMultiple Regression Model: Example

Oil (Gal) Temp Insulation275.30 40 3363.80 27 3164.30 40 1040.80 73 694.30 64 6

230.90 34 6366.70 9 6300.60 8 10237.80 23 10121.40 63 331.40 65 10

203.50 41 6441.10 21 3323.00 38 352.50 58 10

(0F)

Develop a model for estimating Develop a model for estimating heating oil used heating oil used for a single family for a single family home in the month of January based home in the month of January based on on average temperature average temperature and and amount amount of insulation in inches.of insulation in inches.


1 2ˆ 562.151 5.437 20.012i i iY X X

Sample Multiple Regression Equation: Sample Multiple Regression Equation: ExampleExample

CoefficientsIntercept 562.1510092X Variable 1 -5.436580588X Variable 2 -20.01232067

Excel Output

For each degree increase in temperature, the estimated average amount of heating oil used is decreased by 5.437 gallons, holding insulation constant.

For each increase in one inch of insulation, the estimated average use of heating oil is decreased by 20.012 gallons, holding temperature constant.

0 1 1 2 2i i i k kiY b b X b X b X


Venn Diagrams and Venn Diagrams and Explanatory Power of RegressionExplanatory Power of Regression

Oil

Temp

Variations in oil explained by temp or variations in temp used in explaining variation in oil

Variations in oil explained by the error term

Variations in temp not used in explaining variation in Oil SSE

SSR



Oil

Temp

2

r

SSR

SSR SSE

(continued)



Oil

TempInsulation

Overlapping Overlapping variation in both Temp and Insulation are used in explaining the variationvariation in Oil but NOTNOT in the estimationestimation of nor

12

Variation NOTNOT explained by Temp nor Insulation SSE


Coefficient of Coefficient of MultipleMultiple Determination Determination

Proportion of total variation in Y explained by all X variables taken together

Never decreasesNever decreases when a new when a new XX variable variable is added to modelis added to model Disadvantage when comparing models

212

Explained Variation

Total VariationY k

SSRr

SST



Oil

TempInsulation

212

Yr

SSR

SSR SSE


AdjustedAdjusted Coefficient of Multiple Coefficient of Multiple DeterminationDetermination

Proportion of variation in Y explained by all X variables adjusted for the number of X variables used

Penalize excessive use of independent variables

Smaller than Useful in comparing among models

2 212

11 1

1adj Y k

nr r

n k

212Y kr


Coefficient of Multiple DeterminationCoefficient of Multiple Determination

Regression StatisticsMultiple R 0.982654757R Square 0.965610371Adjusted R Square 0.959878766Standard Error 26.01378323Observations 15

Excel Output

SST

SSRr ,Y 2

12

Adjusted r2

reflects the number of explanatory variables and sample size

is smaller than r2


Interpretation of Coefficient of Interpretation of Coefficient of Multiple DeterminationMultiple Determination

96.56% of the total variation in heating oil 96.56% of the total variation in heating oil can be explained by difference in can be explained by difference in temperature and amount of insulationtemperature and amount of insulation

95.99% of the total fluctuation in heating oil can 95.99% of the total fluctuation in heating oil can be explained by difference in temperature and be explained by difference in temperature and amount of insulation after adjusting for the amount of insulation after adjusting for the number of explanatory variables and sample sizenumber of explanatory variables and sample size

2,12 .9656Y

SSRr

SST

2adj .9599r


Using The Model to Make PredictionsUsing The Model to Make Predictions

Predict the amount of heating oil used for Predict the amount of heating oil used for a home if the average temperature is 30a home if the average temperature is 300 0

and the insulation is six inches.and the insulation is six inches.

The predicted heating oil used is 278.97 gallons

1 2

ˆ 562.151 5.437 20.012

562.151 5.437 30 20.012 6

278.969

i i iY X X


Residual PlotsResidual Plots

Residuals vs. May need to transform May need to transform YY variable variable

Residuals vs. May need to transform variableMay need to transform variable

Residuals vs. May need to transform variableMay need to transform variable

Residuals vs. time May have autocorrelationMay have autocorrelation

Y

1X

2X1X

2X


Residual Plots: ExampleResidual Plots: Example

Insulation Residual Plot

0 2 4 6 8 10 12

No discernable patternNo discernable pattern

Temperature Residual Plot

-60

-40

-20

0

20

40

60

0 20 40 60 80

Re

sid

ua

ls

May be some non-May be some non-linear relationshiplinear relationship


Influence AnalysisInfluence Analysis

To determine observations that have influential To determine observations that have influential effect on the fitted modeleffect on the fitted model

Potentially influential points become candidates Potentially influential points become candidates for removal from the modelfor removal from the model

Criteria used areCriteria used are The hat matrix elements The hat matrix elements hhii

The Studentized deleted residuals The Studentized deleted residuals ttii**

Cook’s distance statistic Cook’s distance statistic DDii

All three criteria are complementaryAll three criteria are complementary Only whenOnly when all three criteria all three criteria provide provide

consistent results should an observation be consistent results should an observation be removedremoved


The Hat Matrix Element The Hat Matrix Element hhii

If , Xi is an Influential Point Xi may be considered a candidate for

removal from the model

2

2

1

1 i

i n

ii

X Xh

n X X

2 1 /ih k n


The Hat Matrix Element The Hat Matrix Element hhii : :Heating Oil ExampleHeating Oil Example

Oil (Gal) Temp Insulation h i

275.30 40 3 0.1567363.80 27 3 0.1852164.30 40 10 0.1757

40.80 73 6 0.246794.30 64 6 0.1618

230.90 34 6 0.0741366.70 9 6 0.2306300.60 8 10 0.3521237.80 23 10 0.2268121.40 63 3 0.2446

31.40 65 10 0.2759203.50 41 6 0.0676441.10 21 3 0.2174323.00 38 3 0.1574

52.50 58 10 0.2268

No No hhii > > 0.40.4 No observation appears to No observation appears to be a candidate for removal be a candidate for removal from the modelfrom the modelNOTE: 2(2+1)/ 15= 6/15=NOTE: 2(2+1)/ 15= 6/15=0,40,4

15 2

2 1 / 0.4

n k

k n


The Studentized Deleted Residuals The Studentized Deleted Residuals ttii**

: difference between the observed difference between the observed and and predicted based on a model that includes predicted based on a model that includes all observations except observation all observations except observation ii

: standard error of the estimate for a model that includes all observations except observation i

An observation is considered influential if An observation is considered influential if is the critical value of a two-tail test at is the critical value of a two-tail test at

10% level of significance10% level of significance

* 1

i

i

ii

et

S h

iY

iY

iS

ie

*2i n kt t

2n kt


The Studentized Deleted Residuals The Studentized Deleted Residuals ttii

** :Example :Example

Oil (Gal) Temp Insulation t i*

275.30 40 3 -0.3772363.80 27 3 0.3474164.30 40 10 0.824340.80 73 6 -0.187194.30 64 6 0.0066

230.90 34 6 -1.0571366.70 9 6 -1.1776300.60 8 10 -0.8464237.80 23 10 0.0341121.40 63 3 -1.853631.40 65 10 1.0304

203.50 41 6 -0.6075441.10 21 3 2.9674323.00 38 3 1.168152.50 58 10 0.2432

2 11

15 2

1.7957n k

n k

t t

t10* and t13

* are influential points for potential removal from the model

*10t

*13t


Cook’s Distance Statistic Cook’s Distance Statistic DDii

is the Studentized residualis the Studentized residual

If , an observation is considered is considered influentialinfluential

is the critical value of the is the critical value of the FF distribution at a 50% level of significance distribution at a 50% level of significance

2

2 1i i

ii

SR hD

h

1i

i

YX i

eSR

S h

1, 1i k n kD F

1, 1k n kF


Cook’s Distance Statistic Cook’s Distance Statistic DDii : Heating Oil : Heating Oil ExampleExample

Oil (Gal) Temp Insulation D i

275.30 40 3 0.0094363.80 27 3 0.0098164.30 40 10 0.049640.80 73 6 0.004194.30 64 6 0.0001

230.90 34 6 0.0295366.70 9 6 0.1342300.60 8 10 0.1328237.80 23 10 0.0001121.40 63 3 0.308331.40 65 10 0.1342

203.50 41 6 0.0094441.10 21 3 0.4941323.00 38 3 0.082452.50 58 10 0.0062

No Di >> 0.835 No observation No observation appears to be candidate appears to be candidate for removal from the for removal from the modelmodelUsing the three criteria, there Using the three criteria, there is insufficient evidence for the is insufficient evidence for the removal of any observation removal of any observation from the modelfrom the model

1, 1 3,12

15 2

0.835k n k

n k

F F


Testing for Overall SignificanceTesting for Overall Significance

Show if there is a linear relationship between Show if there is a linear relationship between all of the all of the XX variables together and variables together and YY

Use F test statisticUse F test statistic Hypotheses:Hypotheses:

HH00: : ……kk = = 0 (no linear relationship)0 (no linear relationship) HH11: at least one : at least one ii ( at least one independent ( at least one independent

variable affects variable affects Y Y )) The null hypothesis is a very strong statementThe null hypothesis is a very strong statement Almost always reject the null hypothesisAlmost always reject the null hypothesis


Testing for Overall SignificanceTesting for Overall Significance

Test statistic:

where F has p numerator and (n-p-1) denominator degrees of freedom

(continued)

all /

all

SSR pMSRF

MSE MSE


Test for Overall SignificanceTest for Overall SignificanceExcel Output: ExampleExcel Output: Example

ANOVAdf SS MS F Significance F

Regression 2 228014.6 114007.3 168.4712 1.65411E-09Residual 12 8120.603 676.7169Total 14 236135.2

p = 2, the number of explanatory variables n - 1

p value

Test StatisticMSR

FMSE


Test for Test for OverallOverall Significance SignificanceExample SolutionExample Solution

F0 3.89

H0: 1 = 2 = … = p = 0

H1: At least one i 0 = .05df = 2 and 12

Critical Value(s):

Test statistic:

Decision:

Conclusion:

Reject at = 0.05

There is evidence that There is evidence that at least one independent at least one independent variable affects Yvariable affects Y

= 0.05

F 168.47(Excel Output)


Test for Significance: Test for Significance: IndividualIndividual Variables Variables

Show whether there is a linear Show whether there is a linear relationship between the variable relationship between the variable XXii and and YY

Use tt Test Statistic Hypotheses:Hypotheses:

H0: i 0 (No linear relationship) H1: i 0 (Linear relationship between Xi and

Y)


““T” Test StatisticT” Test StatisticExcel Output: ExampleExcel Output: Example

Coefficients Standard Error t StatIntercept 562.1510092 21.09310433 26.65093769X Variable 1 -5.436580588 0.336216167 -16.16989642X Variable 2 -20.01232067 2.342505227 -8.543127434

t Test Statistic for X1 (Temperature)

t Test Statistic for X2 (Insulation)

i

i

b

btS


t t Test : Example Solution Test : Example Solution

HH00: : 1 1 = 0 = 0

HH11: : 11 0 0

df = 12

Critical Value(s):

Test Statistic:

Decision:

Conclusion:

Reject Reject HH00 at at = 0.05 = 0.05

There is evidence of a There is evidence of a significant effect of significant effect of temperature on oil temperature on oil consumption.consumption.t0 2.1788-2.1788

.025

Reject H0 Reject H0

.025

Does temperature have a significant effect on monthly Does temperature have a significant effect on monthly consumption of heating oil? Test at consumption of heating oil? Test at = 0.05 = 0.05.

tt Test Statistic = -16.1699 Test Statistic = -16.1699


Venn Diagrams and Venn Diagrams and Estimation of Regression ModelEstimation of Regression Model

Oil

TempInsulation

Only this information Only this information is used in the is used in the estimation of estimation of

2

Only this Only this information is information is used in the used in the estimation ofestimation of

1This This information is information is NOT used in NOT used in the the estimation of estimation of nor nor1 2


Confidence Interval Estimate Confidence Interval Estimate for the Slopefor the Slope

Provide the 95% confidence interval for the population slope 1 (the effect of temperature on oil consumption).

11 1n p bb t S

Coefficients Lower 95% Upper 95%Intercept 562.151009 516.1930837 608.108935X Variable 1 -5.4365806 -6.169132673 -4.7040285X Variable 2 -20.012321 -25.11620102 -14.90844

-6.169 1 -4.704

The estimated average consumption of oil is reduced by between 4.7 gallons to 6.17 gallons per each increase of 10 F.


Contribution of a Single Independent Contribution of a Single Independent VariableVariable

Let Xk be the independent variable of be the independent variable of interestinterest

Measures the contribution of Measures the contribution of XXkk in in explaining the total variation in explaining the total variation in YY ( (SSTSST))

kX

| all others except

all all others except

k k

k

SSR X X

SSR SSR X


Contribution of a Single Independent Variable kX

1 2 3

1 2 3 2 3

| and

, and and

SSR X X X

SSR X X X SSR X X

Measures the contribution of in explaining SST

1X

From ANOVA section of regression for


0 1 1 2 2 3 3i i i iY b b X b X b X 0 2 2 3 3i i iY b b X b X


Coefficient of Partial Determination ofCoefficient of Partial Determination of

Measures the proportion of variation in Measures the proportion of variation in the dependent variable that is explained the dependent variable that is explained by by XXkk while controlling for (holding while controlling for (holding constant) the other independent constant) the other independent variables variables

2 all others

| all others

all | all others

Yk

k

k

r

SSR X

SST SSR SSR X

kX


Coefficient of Partial Determination forCoefficient of Partial Determination forkX

(continued)

1 221 2

1 2 1 2

|

, |Y

SSR X Xr

SST SSR X X SSR X X

Example: Two Independent Variable Model


Venn Diagrams and Coefficient of Partial Venn Diagrams and Coefficient of Partial Determination forDetermination for

kX

Oil

TempInsulation

1 2|SSR X X

21 2

1 2

1 2 1 2

|

, |

Yr

SSR X X

SST SSR X X SSR X X

=


Contribution of a Subset of Contribution of a Subset of Independent VariablesIndependent Variables

Let Let XXss be the subset of independent be the subset of independent variables of interestvariables of interest

Measures the contribution of the subset Measures the contribution of the subset xxss in in explaining explaining SSTSST

| all others except

all all others except

s s

s

SSR X X

SSR SSR X


Contribution of a Subset of Independent Contribution of a Subset of Independent Variables: ExampleVariables: Example

Let Let XXss be be XX11 and and XX33

1 3 2

1 2 3 2

and |

, and

SSR X X X

SSR X X X SSR X



0 1 1 2 2 3 3i i i iY b b X b X b X 0 2 2i iY b b X


Testing Portions of ModelTesting Portions of Model

Examines the contribution of a Examines the contribution of a subset subset XXss of explanatory variables of explanatory variables to the relationship with to the relationship with YY

Null hypothesisNull hypothesis: Variables in the subset do not

significantly improve the model when all other variables are included

Alternative hypothesisAlternative hypothesis: At least one variable is significant


Testing Portions of ModelTesting Portions of Model

Always one-tailed rejection regionone-tailed rejection region Requires comparison of two regressions

One regression includes everything Another regression includes everything

except the portion to be tested

(continued)


Partial Partial FF Test For Contribution of Test For Contribution of Subset of Subset of XX variables variables

HypothesesHypotheses: H0 : Variables Xs do not significantly improve

the model given all others variables included H1 : Variables Xs significantly improve the

model given all others included Test StatisticTest Statistic:

with df = m and (n-p-1) m = # of variables in the subset Xs

| all others /

allsSSR X m

FMSE


Partial Partial FF Test For Contribution of A Single Test For Contribution of A Single

HypothesesHypotheses: H0 : Variable Xj does not significantly improve

the model given all others included H1 : Variable Xj significantly improves the

model given all others included Test StatisticTest Statistic:

With df = 1 and (n-p-1) m = 1 here

jX

| all others

alljSSR X

FMSE


Testing Portions of Model: ExampleTesting Portions of Model: Example

Test at the Test at the = .05 level to = .05 level to determine whether the determine whether the variable of average variable of average temperature significantly temperature significantly improves the model given improves the model given that insulation is includedthat insulation is included.


Testing Portions of Model: ExampleTesting Portions of Model: Example

H0: X1 (temperature) does not improve model with X2 (insulation) included

H1: X1 does improve model

= .05, df = 1 and 12

Critical Value = 4.75

ANOVASS

Regression 51076.47Residual 185058.8Total 236135.2

ANOVASS MS

Regression 228014.6263 114007.313Residual 8120.603016 676.716918Total 236135.2293

(For X1 and X2) (For X2)

Conclusion: Reject H0; X1 does improve model

1 2

1 2

| 228,015 51,076261.47

, 676.717

SSR X XF

MSE X X


When to Use the When to Use the F testF test

The The FF test for the inclusion of a single test for the inclusion of a single variable after all other variables are variable after all other variables are included in the model is IDENTICAL to included in the model is IDENTICAL to the the tt test of the slope for that variable test of the slope for that variable

The only reason to do an The only reason to do an F F test is to test test is to test several variables togetherseveral variables together

Multiple Regression

Documents

Transcript of Multiple Regression