8/13/2019 2 Linear Model [Final]
1/59
DrMudassirUddin
DrMudassirUddin
Review of Linear Model Basics
8/13/2019 2 Linear Model [Final]
2/59
Linear Regression Lecture [1]
Gauss-Markov Assumptions for Classical Linear Regression
Functional Form: Y
(n1)= X
(nk)(k1)+
n1(Xis full rank and has leading column of 1s)
Mean Zero Errors: E[|X] =0Homoscedasticity: Var[] =2I
Non-Correlated Errors: Cov[i, j] = 0, i =j
Exogeneity of Explanatory VariablesCov[i|X] = 0, i
8/13/2019 2 Linear Model [Final]
3/59
Linear Regression Lecture [2]
Other Considerations
Requirements: conformability, Xis rankk
Freebee: eventual normality. . . |X N(0, 2I)
8/13/2019 2 Linear Model [Final]
4/59
Linear Regression Lecture [3]
Least Squares Estimation (OLS)
Define the following function:
S() == (Y X)(Y X)= YY
(1n)(n1) 2YX
(1n)(nk)(k1)+ XX
(1k)(kn)(nk)(k1)
Take the derivative ofS() with respect to :
S() = 0 2 XY
(kn)(n1)+ 2XX
(kn)(nk)(k1) 0
So there exists a solution at some value of: XXb= XYwhich is theNormal Equation
Premultiplying the Normal Equation by (XX)1, gives: = (XX)1XY, where we can call as bfor notational convenience (this is where the requirement for XXto be nonsingular comesfrom)
8/13/2019 2 Linear Model [Final]
5/59
Linear Regression Lecture [4]
Implications
Normal Equation: XXbXY= X(Y Xb) = Xe 0(by assumption)
ei 0
the regression hyperplane pass through means: Y= Xb
mean(Y) =mean(Y)
the hat matrix (H, P, or (IM)):e= Y Xb
=Y X((XX)1XY)=Y (X(XX)1X)Y=Y HY= (IH)Y=MY
whereMis symmetric and idempotent.
8/13/2019 2 Linear Model [Final]
6/59
Linear Regression Lecture [5]
The HAT Matrix
the name is because Y = Xb= X((XX)
1
XY) = (X(XX)1
X)Y = HY, but projectionmatrix is better for geometric reasons.
properties of interest:
IM= P, IP= M PX= X
PM= MP = 0 (orthogonality)
ee= Y MMY= Y MY= Ye(sum of squares)
Using the Normal Equation, XXb= XY,
ee= (Y Xb)(Y Xb) =Y Y YXb bXY+ bXXb
=Y Y (XXb)b b(XXb)+ bXXb=Y Y (bX)(Xb)=Y Y YY
8/13/2019 2 Linear Model [Final]
7/59
Linear Regression Lecture [6]
Fit & Decomposition
Definitions:
SST =n
i=1
(Yi Y)2
SSR =n
i=1(Yi Y)2
SSE =n
i=1
(Yi Yi)2
8/13/2019 2 Linear Model [Final]
8/59
Linear Regression Lecture [7]
Fit & Decomposition
Interesting manipulations of the sum of squares total:
SST =n
i=1
(Y2i 2YiY+ Y2)
=n
i=1
Y2i 2n
i=1
YiY+nY2
=n
i=1
Y2i 2nY2 +nY2
=n
i=1
Y2i nY2 (scalar description)
=n
i=1 Y2i
n
1
n
n
i=1 Yi2
=Y Y 1nYJY (matrix algebra description)
whereJis an nmatrix of all 1s.
8/13/2019 2 Linear Model [Final]
9/59
Linear Regression Lecture [8]
More Decomposition
Sum of Squares Regression:
SSR =n
i=1
(Y2i 2YiY+ Y2)
= YY 2Yn
i=1Yi+nY
2
= (bX)Xb 2nY2 +nY2
= (bX)Xb nY2
=bXY 1nYJY
8/13/2019 2 Linear Model [Final]
10/59
Linear Regression Lecture [9]
Sum of Squares Error:
SSE =
n
i=1
(Yi Yi)2
=ee
= (Y X)(Y X)=Y Y YXb bXY+ bXXb=Y Y
bXY+ (bX
Y)Xb
=Y Y bXY+ (0)Xb=Y Y bXY
8/13/2019 2 Linear Model [Final]
11/59
Linear Regression Lecture [10]
Magic
SSR + SSE:SS R+SS T = (bXY 1
nYJY) + (YY bXY)
=Y Y 1nYJY
= SST
8/13/2019 2 Linear Model [Final]
12/59
Linear Regression Lecture [11]
A Measure of Fit
The R-Square or R-Squared measure:
R2 =SSR
SS T =
SST SSESST
= 1 ee
YMoY=bXMoXbYMoY
whereMo =I 1nii, i= c(1, 1, . . . , 1).Note: Mo is idempotent and transforms means to deviances for the explanatory variables.
Warning: R2 is not a statistic and does not have quite the meaning that some expect.
There is another version that accounts for sample size and the number of explanatory variables:
R2adj = ee/(n k)YMoY/(n 1)
8/13/2019 2 Linear Model [Final]
13/59
Linear Regression Lecture [12]
Properties of OLS
bis unbiased for :b= (XX)1XY
= (XX)1X(X+)
= (XX)1XX+ (XX)1X
= + (XX)1X
SoE[b] =.
8/13/2019 2 Linear Model [Final]
14/59
Linear Regression Lecture [13]
Properties of OLS
The variance ofbis:Var[b] =E[(b )](b )] (E[b ])(E[b ])
=E[(b )](b )]=E[((XX)1X)((XX)1X)]
= (XX)1XE[]X(XX)1
= (XX)1X(2I)X(XX)1
=2(XX)1
8/13/2019 2 Linear Model [Final]
15/59
Linear Regression Lecture [14]
Properties of OLS
Given Gauss-Markov assumptions, bis BLUE for
if calculated from OLS. b|X N(, 2(XX)1).Est.Var[b] =s2(XX)1, wheres2 =ee/(n k), the standard error of the regression.Most econometric texts make a distinction between non-stochastic explanatory variables (those
set in experimental settings) and stochastic explanatory variables which are set by nature/survey/others.
Perspective here: Xis fixed once observed and is the random variable:
since Var[] =I, then no single term dominates and we get the Lindberg-Feller CLT result.
so (the sample quantity) is IID normal and we write the joint PDF as:
f() =
n
i=1 f(i) = (22)n2 exp[ee/22](estimation fromE[ee] =2).
8/13/2019 2 Linear Model [Final]
16/59
Linear Regression Lecture [15]
Estimating From Sample Quantitities
Population derived variance/covariance matrix: Var[b
] =
2
(XX
)1
.We also know: E[ei] =i.
And: E[e2i ] = Var[i] + (E[i])2 =tr(2I) =n2.
So why not use: 2 1n
e2i .
But:
ei= yi Xib (now insert population values)= (Xi+i) Xib=i Xi(b )
whereE[ei] =i.
8/13/2019 2 Linear Model [Final]
17/59
Linear Regression Lecture [16]
Estimating From Sample Quantitities
Recall that:M
=IH
=IX
(XX
)1X
. So that:My= (IX(XX)1X)y
=y X(XX)1Xy=y Xb=e
So e= Me since e= y y.And therefore:
My= M[X+]
=MX+ M
= (I
X(XX)1)X+ M
=X X(XX)1)XX+ M=M
So e= Me and ee= (M)M= MM= M.
8/13/2019 2 Linear Model [Final]
18/59
Linear Regression Lecture [17]
Estimating From Sample Quantitities
So we can use this:
E[ee|X] =E[eMe|X]=E[tr(eMe)|X] (Gauss-Markov assumption)=E[tr(Mee)|X] (properties of traces: tr(ABC) = tr(CBA)= tr(ME[ee|X]) (M is fixed for observedX= tr(M)2
= [tr(I) tr(X(XX)1X)]2= [n k]2
Tricks:
tr[X(XX)1X] = tr[(XX)1XX] =k
rank[A] = tr[A] for symmetric idempotentArank[ABC] = rank[B] ifA, B nonsingular
so tr[H] = rank[X] = rank[X] =k
8/13/2019 2 Linear Model [Final]
19/59
Linear Regression Lecture [18]
Estimating From Sample Quantitities
From the biased estimator of2,E[ee|X] = (n
k)2, we get:
2 = een k =s
2,
so that an estimator of Var[b] is:
Var[b] =s2(XX)1.
This sets up Wald-style traditional linear inference:
zk= bk nullk
2(XX)1asym. N(0, 1),
provided that we know2 (which we usually do not).
But we know that:
(n k)s22
2nk=X2
zk/
X2/df t(nk)(0)if the random variableszk andX
2 are independent.
8/13/2019 2 Linear Model [Final]
20/59
Linear Regression Lecture [19]
Estimating From Sample Quantitities
Making the obvious substitution gives:
t(nk)= bk nullk
2(XX)1 1
(nk)s22(nk)
= bk nullk
s2(XX)1
Typical (Wald) regression test:
H0 : k= 0 H1 : k= 0
making:
t(nk)= bk nullk
s2(XX)1= bk
SE(k)
Alternatives usually look like:
H0 : k
8/13/2019 2 Linear Model [Final]
21/59
Linear Regression Lecture [20]
Summary Statistics
(1
) Confidence Interval for bk:bk SE(b)t/2,df:bk+SE(b)t/2,df
(1 ) Confidence Interval for2:
(n k)s22/2
:(n k)s2
21
/2
F-statistic test for all but b0zero:
F = SSR/(k 1)SSE/(n k) Fk1,nkunder the null
8/13/2019 2 Linear Model [Final]
22/59
Linear Regression Lecture [21]
Multicollinearity Issues
If one explanatory variable is a linear combination of another then rank(X) =k
1.
Therefore rank(XX) =k 1 (matrix sizek k), and it is singular and non-invertible.Now no parameter estimates are possible, and model is now unidentified.
More typically: 2 explanatory variables are highly but not perfectly correlated.
Symptoms: small changes in data give large changes in parameter estimates.
coefficients have large standard errors and poort-statistics even if F-statistics andR2 are okay.
coefficients seem illogical (wrong sign, huge magnitude)
8/13/2019 2 Linear Model [Final]
23/59
Linear Regression Lecture [22]
Multicollinearity Remedies
respecify model (if reasonable)
center explanatory variables, or standardize
ridge regression (add a little bias):
b= [XX+RI]1Xy
such that the [ ] part barely inverts.
8/13/2019 2 Linear Model [Final]
24/59
Linear Regression Lecture [23]
Is bUnbiased?
Starting with b:
b= (XX)1Xy= (XX)1X(X+) = + (XX)1X,
and take expected values:
E[b] =E[+ (XX)1X] =E[] +E[(XX)1X] = +E[K] =
8/13/2019 2 Linear Model [Final]
25/59
Linear Regression Lecture [24]
What is the Variance ofb?
By definition:
Var[b|X] =E[(b )(b )|X] E[(b |X)]2.Since b = (XX)1X,
Var[b|X] =E((XX)1X)((XX)1X)|X=E
(XX)1XX(XX)1|X
= (XX)1XE[ee|X]X(XX)1= (XX)1X2IX(XX)1
=2(XX)1XX(XX)1
=2(XX)1
Li R i L [ ]
8/13/2019 2 Linear Model [Final]
26/59
Linear Regression Lecture [25]
Testing Linear Restrictions
A theory has testable implicationsif it implies some testable restrictions on the model definition:
H0 : k= 0 versus H1 : k= 0for example.
Most restrictions involve nestedparameter space:
unrestricted:[0, 1, 2, 3]
restricted:[0, 0, 2, 3].
Note that non-nested comparisons cause problems for non-Bayesians.
Li R i L t [26]
8/13/2019 2 Linear Model [Final]
27/59
Linear Regression Lecture [26]
Testing Linear Restrictions
Linear restrictions for regression are clear when:
r111+r122+. . .+r1kk=q1
r211+r222+. . .+r2kk=q2...
rj11+rj22+. . .+rjkk=qj
or in more succinct matrix algebra form: R(Jk)
= q.
Notes:
Each row ofR is one restriction.
J < k for R to be full rank.
This setup imposes J restrictions onk parameters, so there arek J free parameters left. We are still assuming that i N(0, 2)
General test:
H0 : R q= 0 H1 : R q =0
Linear Regression Lecture [27]
8/13/2019 2 Linear Model [Final]
28/59
Linear Regression Lecture [27]
Testing Linear Restrictions, Examples
One of the coefficients is zero, j = 0,J= 1:
R= [0, 0, 0, 1j
, . . . , 0, 0, 0], q= 0
Two coefficients are equal,j =k,J= 2:
R= [0, 0, 0, 1j
, . . . ,1k
, . . . , 0, 0, 0], q= 0
Three coefficients are zero, J= 3: 1 0 0 0 . . . 00 1 0 0 . . . 00 0 1 0 . . . 0
= [I:0], q= 00
0
Linear Regression Lecture [28]
8/13/2019 2 Linear Model [Final]
29/59
Linear Regression Lecture [28]
Testing Linear Restrictions, Examples
A set of coefficients sum to one, 2+3+4= 1:
R= [0, 1, 1, 1, 0, . . . , 0], q= 1
Several restrictions,2+3= 4,4+6= 0,5= 9:
R=
0 1 1 0 0 0
0 0 0 1 0 1
0 0 0 0 1 0
, q=
4
0
9
All coefficients except the constant are zero:
R= [0:I], q= [0]
Linear Regression Lecture [29]
8/13/2019 2 Linear Model [Final]
30/59
Linear Regression Lecture [29]
Testing Linear Restrictions, Examples
Define now the discrepency vectordictated by the null hypothesis:
Rb q= m 0which asks whether mis sufficiently different from zero. Note that mis linear function ofband
therefore also normally distributed.
This makes it easy to think about:
E[m|X] =RE[b|X] q= R q= 0Var[m|X] = Var[Rb q|X] =R[Var[b|X]R= 2R(XX)1R
Wald Test:
W =m[Var[m|X]]1m= (Rb q)[2R(XX)1R]1(Rb q) 2JwhereJis the number of rows ofR, i.e. the number of restrictions.
Linear Regression Lecture [30]
8/13/2019 2 Linear Model [Final]
31/59
Linear Regression Lecture [30]
Testing Linear Restrictions, Examples
Unfortunately we do not have 2, so we use the test:
F =2(Rb q)(R(XX)1R)(Rb q) 12J
2
s2
n kn k
=
(Rb q)(2R(XX)1R)1(Rb q)/J[(n k)s2/2]/(n k)
= Xn
2J/J
Xd 2nk/(n k) Fj,nk =thats the distributional interpretation, now simplify:
= 1
J(Rb q)(s2R(XX)1R)1(Rb q)
Linear Regression Lecture [31]
8/13/2019 2 Linear Model [Final]
32/59
g [ ]
Testing Linear Restrictions, Examples
Example with 1 linear restriction, start with the definition:
H0 : r11+r22+. . .+rkk=r= q
F1,nk=
j(rjbj q)2
j
krjrkEst.Cov.[bj, bk]
and be more specific:
= 0, so R= [0, 0, . . . , 0, 1, 0, . . . , 0, 0],q= [0]
so that R(XX)1Rsimplifies to thej th diagonal of (XX)1. Now:
Rb q= bj q, F1,nk = bj qEst.V ar.[bj]
12
.
Linear Regression Lecture [32]
8/13/2019 2 Linear Model [Final]
33/59
Non-Normal Errors
Withouti
N(0, 2) we do not have F,t,2 results (bummer).
Despite that, we knowthat the asymptotic disribution ofbis:
b N,
2
nQ1
whereQ= plim
XX
n and plims2 =2 wheres2 = e
enk .
So the test statistic is:
tk=
n(bk nullk )s2(XX/n)1
tnk
provided that ei N(0, 2).Since the denominator converges to
2Q1, then:
k=n(bk nullk )
2Q1=
n(bk nullk )Asym.V ar.[bk]
1
2
thus asymptotically justifying at-test without the assumption of normality.
Linear Regression Lecture [33]
8/13/2019 2 Linear Model [Final]
34/59
Non-Normal Errors Summary
Major theorem, if:
n(b ) dN(0, 2Q1)
and if
H0 : R q= 0is true, then
W = (Rbq)[Rs2(XX)1R]1(Rb
q) =J F d
2
J
Linear Regression Lecture [34]
8/13/2019 2 Linear Model [Final]
35/59
Testing Nonlinear Restriction
H0 :c() =qwherec() is some nonlinear function.
Simple 1-restriction case:
z= c()
est.SE tnk
(or equivalentlyz2 F1,nk).But getting est.SE is hard, so use the first two terms of a Taylor series expansion to get an
estimate:
f(b) =f(a) +f(a)(b 1)1
1! +f(a)
(b a)22!
+f(a)(b a)3
3! +. . .
meaning:
c() c() +
c()
( )
so plim= justifies usingc() instead ofc().
Linear Regression Lecture [35]
8/13/2019 2 Linear Model [Final]
36/59
Testing Nonlinear Restriction
Now we can calculate the needed variance term:
Var(c() =E
c()2 (E[c()])2
=E
c() +
c()
( )
(E[c()])2
=Ec()2 2c()c()
(
) + c()
(
)2c()
(E[c()])2=q2 2q
c()
(0) +E
c()
( )2
c()
q2
=E
c()
Var()
c()
sinceE( )2 =E()2 2 = Var().All this means that we can use sample estimates forc()/and plug ins2(XX)1 for Var()
and then test with a normal distribution.
Linear Regression Lecture [36]
8/13/2019 2 Linear Model [Final]
37/59
Linear Model Predictions
We want the predicted value for x0 not in the sample:
y0 =x0+0 y0 =x0b
since y0 is the LMVUE ofE[y0|x0].The prediction error is:
e0 =y0 y0 =x0( b) +0.The Prediction variance is:
Var[e0|X,x0] =2 + Var[x0( b)|X,x0] =2 + x02(XX)1x0
and if we have a constant term in the regression, this is equivalent to:
21 +1n+K1
j=1K1
k=1 (x0j xj)(x0k xk)(X1M0X1)jk ,where X1 is Xomitting the first column, Kis the number of explanatory variables (includingthe constant), and M0 =I 1nii.
Linear Regression Lecture [37]
8/13/2019 2 Linear Model [Final]
38/59
Linear Model Predictions
This shows that the variance is greater the further away x0 is from x. Classic diagram: where
prediction intervals are modeled y0 t/2Var(e0)
Linear Regression Lecture [38]
8/13/2019 2 Linear Model [Final]
39/59
Running Lowess
postscript("Class.Multilevel/trends1.ps")par(bg="lightgrey")
x
8/13/2019 2 Linear Model [Final]
40/59
Running Lowess
+
+
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+++
+
+
+
++
+
+
++
+
+++
+
+
++
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
++
+
+
++
+
+
+++
+
+
+
+
++
+
++
+
+
+
+
++
+
+++
++
+
+
+
+++
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
++
+
+
++
+
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+
++
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+++
+
+
+
++
+
+
++
+
+++
+
+
+
+
+
++
+
+
+
+
++
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+
++
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+++
+
+
+
++
+
+
++
+
+++
+
+
++
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+++
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
++
+
+
++
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+
++
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
++
+
+
++
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+
+++
+
+
+
+
+
++
+
+
+
+
+
+
+++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+++
+
++
5 10 15 20 25
0
.2
0.0
0.2
0.4
0.6
0.8
1.0
Linear Regression Lecture [40]
8/13/2019 2 Linear Model [Final]
41/59
Example: Poverty Among the Elderly, Europe
Governments often worry about the economic condition of senior citizens for political and social
reasons.
Typically in a large industrialized society, a substantial portion of these people obtain the bulk of
their income from government pensions.
An important question is whether there is enough support through these payments to provide
subsistence above the poverty rate.
To see if this is a concern, the European Union (EU) looked at this question in 1998 for the (then)
15 member countries with two variables:
1. the median (EU standardized) income of individuals age 65 and older as a percentage of the
population age 064,
2. the percentage with income below 60% of the median (EU standardized) income of the national
population.
Linear Regression Lecture [41]
8/13/2019 2 Linear Model [Final]
42/59
Example: Poverty Among the Elderly, Europe
The data from the European Household Community Panel Survey are:
Relative PovertyNation Income RateNetherlands 93.00 7.00Luxembourg 99.00 8.00Sweden 83.00 8.00Germany 97.00 11.00
Italy 96.00 14.00Spain 91.00 16.00Finland 78.00 17.00France 90.00 19.00United.Kingdom 78.00 21.00Belgium 76.00 22.00Austria 84.00 24.00Denmark 68.00 31.00Portugal 76.00 33.00Greece 74.00 33.00Ireland 69.00 34.00
Linear Regression Lecture [42]
8/13/2019 2 Linear Model [Final]
43/59
Example: Poverty Among the Elderly, Europe
eu.pov
8/13/2019 2 Linear Model [Final]
44/59
Example: Poverty Among the Elderly, Europe
60 70 80 90 100
5
10
15
20
25
30
35
NetherlandsLuxembourgSweden
Germany
Italy
SpainFinland
France
United.Kingdom
Belgium Austria
Denmark
Portugal
GreeceIreland
Relative Income, Over 65
Povertyrate
,O
ver65
Linear Regression Lecture [44]
8/13/2019 2 Linear Model [Final]
45/59
Example: Poverty Among the Elderly, Europe
postscript("Class.Multilevel/trends3.ps")
par(mfrow=c(1,1),mar=c(4,4,2,2),lwd=5,bg="lightgrey")plot(eu.pov,pch=15,xlab="",ylab="",ylim=c(2,37),xlim=c(61,104))
x.y.fit
8/13/2019 2 Linear Model [Final]
46/59
Example: Poverty Among the Elderly, Europe
60 70 80 90 100
5
10
15
20
25
30
35
NetherlandsLuxembourgSweden
Germany
Italy
SpainFinland
France
United.Kingdom
Belgium Austria
Denmark
Portugal
GreeceIreland
Relative Income, Over 65
Povertyrate
,O
ver65
Linear Regression Lecture [46]
8/13/2019 2 Linear Model [Final]
47/59
Example: Poverty Among the Elderly, Europe
summary(x.y.fit)Call:
lm(formula = eu.pov[, 2] ~ eu.pov[, 1])
Residuals:
Min 1Q Median 3Q Max
-12.224 -3.312 1.482 3.923 7.424
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 83.6928 12.2526 6.831 1.21e-05
eu.pov[, 1] -0.7647 0.1458 -5.246 0.000158
Residual standard error: 5.611 on 13 degrees of freedom
Multiple R-Squared: 0.6792, Adjusted R-squared: 0.6545
F-statistic: 27.52 on 1 and 13 DF, p-value: 0.0001580
Linear Regression Lecture [47]
8/13/2019 2 Linear Model [Final]
48/59
More on Linear Models
In this example the slope of the line is
0.7647, which tells us how much the poverty rate changes
for a one unit increase in income.
The intercept is 83.6928, which is what poverty would be for zero income.
The standard error is a measure of how reliable these estimates are. One common rule of thumb
is to see if the standard error is half the coefficient estimates or less.
R-Squared tells us how much of the variance of the outcome variable can be explained by theexplanatory variable.
F-statistic tells us how well significant the model fit is.
Linear Regression Lecture [48]
8/13/2019 2 Linear Model [Final]
49/59
OECD Example
The data are from the Organization for Economic Cooperation and Development (OECD) and
highlight the relationship between
commitment to employment protection measured on an interval scale (0-4) indicating the
quantity and extent of national legislation to protect jobs,
the total factor productivity difference in growth rates between 1980-1990 and 1990-1998.
for 19 countries.
For details, see The Economist, September 23, 2000.
Linear Regression Lecture [49]
8/13/2019 2 Linear Model [Final]
50/59
OECD Example
Prot. Prod.
United States 0.2 0.5Canada 0.6 0.6
Australia 1.1 1.3
New Zealand 1.0 0.4
Ireland 1.0 0.1
Denmark 2.0 0.9
Finland 2.2 0.7Austria 2.4 -0.1
Belgium 2.5 -0.4
Japan 2.6 -0.4
Sweden 2.9 0.5
Netherlands 2.8 -0.5
France 2.9 -0.9
Germany 3.2 -0.2
Greece 3.6 -0.3
Portugal 3.9 0.3
Italy 3.8 -0.3
Spain 3.5 -1.5
Linear Regression Lecture [50]
8/13/2019 2 Linear Model [Final]
51/59
OECD Example
oecd
8/13/2019 2 Linear Model [Final]
52/59
OECD Example
0 1 2 3 4
1
.5
1
.0
0
.5
0.0
0.5
1.0
1.5
UnitedStatesCanada
Australia
NewZealand
Ireland
Denmark
Finland
Austria
Belgium Japan
Sweden
Netherlands
France
GermanyGreece
Portugal
Italy
Spain
Employment Protection Scale
Tota
lFactor
Pro
ductiv
ity
Difference
Linear Regression Lecture [52]
8/13/2019 2 Linear Model [Final]
53/59
OECD Example
oecd.fit |t|)
(Intercept) 0.8591 0.3174 2.706 0.0156
oecd$Prot -0.3496 0.1215 -2.878 0.0109
---Residual standard error: 0.5761 on 16 degrees of freedom
Multiple R-Squared: 0.3411, Adjusted R-squared: 0.2999
F-statistic: 8.284 on 1 and 16 degrees of freedom, p-value: 0.01093
Linear Regression Lecture [53]
8/13/2019 2 Linear Model [Final]
54/59
So What Does a BadRegression Model Look Like?
Run the following....
time = c(0.25, 0.5, 0.75, 1, 1.25, 2, 3, 4, 5, 6, 8, 0.25, 0.5,
0.75, 1, 1.25, 2, 3, 4, 5, 6, 8, 0.25, 0.5, 0.75, 1, 1.25, 2, 3,
4, 5, 6, 8, 0.25, 0.5, 0.75, 1, 1.25, 2, 3, 4, 5, 6, 8, 0.25,
0.5, 0.75, 1, 1.25, 2, 3, 4, 5, 6, 8, 0.25, 0.5, 0.75, 1, 1.25,
2, 3, 4, 5, 6, 8)
conc = c(1.5, 0.94, 0.78, 0.48, 0.37, 0.19, 0.12, 0.11,
0.08, 0.07, 0.05, 2.03, 1.63, 0.71, 0.7, 0.64, 0.36, 0.32, 0.2,0.25, 0.12, 0.08, 2.72, 1.49, 1.16, 0.8, 0.8, 0.39, 0.22, 0.12,
0.11, 0.08, 0.08, 1.85, 1.39, 1.02, 0.89, 0.59, 0.4, 0.16, 0.11,
0.1, 0.07, 0.07, 2.05, 1.04, 0.81, 0.39, 0.3, 0.23, 0.13, 0.11,
0.08, 0.1, 0.06, 2.31, 1.44, 1.03, 0.84, 0.64, 0.42, 0.24, 0.17,
0.13, 0.1, 0.09)
#postscript("Class.Multilevel/conc.ps"); par(bg="lightgrey")
conc.fit
8/13/2019 2 Linear Model [Final]
55/59
Extended Policy Example
# SIMPLE REGRESSION EXAMPLE USING THE DETROIT MURDER DATASET, YEARS 1961-1974# R CODE HERE RUNS A BASIC MODEL AND VARIOUS DIAGNOSTICS, EMAIL QUESTIONS
#FTP - Full-time police per 100,000 population
#UEMP - % unemployed in the population
#MAN - number of manufacturing workers in thousands#NMAN - Number of non-manufacturing workers in thousands
#GOV - Number of government workers in thousands
#LIC - Number of handgun licences per 100,000 population (you can buy one)
#GR - Number of handgun registrations per 100,000 population (you own one)
#CLEAR - % homicides cleared by arrests
#WM - Number of white males in the population
#HE - Average hourly earnings
#WE - Average weekly earnings
Linear Regression Lecture [55]
8/13/2019 2 Linear Model [Final]
56/59
#HOM - Number of homicides per 100,000 of population
#ACC - Death rate in accidents per 100,000 population
#ASR - Number of assaults per 100,000 population
# LOAD DATA FILE, CREATE DATA FRAME
count.fields("public_html/data/detroit.data")
count.fields("http://jgill.wustl.edu/data/etroit.data")
detroit.df
8/13/2019 2 Linear Model [Final]
57/59
# RUN A MODEL, SUMMARIZE THE RESULTS
detroit.ols
8/13/2019 2 Linear Model [Final]
58/59
rstudent(detroit.ols)
dfbetas(detroit.ols)
dffits(detroit.ols)
covratio(detroit.ols)cooks.distance(detroit.ols)
X
8/13/2019 2 Linear Model [Final]
59/59
for (i in 1:length(cooks.vals))
segments(i,0,i,cooks.vals[i])
mtext(side=3,cex=1.3,"Leverage and Influence: Detroit Murder Data",outer=T,line=2)
# dev.off()
# JACKKNIFE OUT THE 8TH CASE AND RERUN MODEL
detroit2.df
Top Related