Applied linear statistical models: An overviewgunnar/kennsla/alsm/alsmintro/al...Applied linear...

ALSM

GS

The course

Simple linearregressionFitting

Some theory

Model evaluation

Multiple linearregressionMatrices

Some more theory

Applied stuff

Other things

Applied linear statistical models: Anoverview

Gunnar Stefansson

1Dept. of MathematicsUniv. Iceland

August 27, 2010

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Outline

The course

Simple linear regressionFittingSome theoryModel evaluation

Multiple linear regressionMatricesSome more theoryApplied stuff

Other things

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Some basics

Course: Applied linear statistical modelsThis lecture: A description of the material to be coveredduring the course and of the framework, where materialis, etc.Course material is at

I Content+quizzes http://tutor-web.net→http://vr3pc109.rhi.hi.is/

I The bookI Other handouts (weekly homeworks, 3-4 projects

(40%), copies from Scheffe etc)I Instructor’s home page: http://www.hi.is/∼gunnar

(data sets,links, this file)I The Univ. Icel. web system (announcements/ email)

Important: Read your e-mail for announcements!Warning: The tutor-web material is under development!

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

The homework

weekly homework, on handouts (possibly within tutorials):

I on-line quizzesI from bookI other exercisesI modify wikipediaI write examples etc for tutor-web

3-4 projects (30%)first handout: tutor-web, “STAT 645 smplreg” – do notprint yetR stuff: tutor-web, “STAT 150 R – do not print yetmath stuff: tutor-web, “MATH 612 ccas

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Outline

The course



Other things

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Fitting a line to data: STATS 310 smplreg

Simple linear regression: Haven pairs (x1, y1), . . . , (xn, yn) andwant ”best” fitting line.Estimation (OLS):

S(α, β) =∑

i

(yi − (α + βxi))2

Minimize S over α, β to get

a = y − bx

b =

∑i(xi − x)(yi − y)∑

i(xi − x)2

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

1500 2000 2500 3000 3500 4000

400

500

600

700

800

900

1000

Capelin biomass

Cod

gro

wth

Example: Cod growth vs capelin

biomass.

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

A formal statistical model

Fixed numbers, xiRandom variables:Yi ∼ n(α + βxi , σ

2)or: Yi = α + βxi + εiwith εi ∼ n(0, σ2) in-dependent and identicallydistributed (i.i.d.)The data:

yi = α + βxi + ei

So: yi -values are out-comes of the random vari-ables Yi , but xi -values areconstants.

0 1 2 3 4 5

05

1015

20

xy

Usual regression assumptions.

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

The assumptions

The assumptions (which may all fail) are:

I x-values are constants (no error)I linearityI constant varianceI GaussianI Independence

Will test these and modify accordinglyExamples: Fish growth (nonlinear); Bird counts(nonnormal); fuel consumption (heteroscedastic); stockprices (autocorrelated)

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Outline

The course



Other things

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Some distributionsUnivariate Gaussian density

f (y) =1√2πσ

e−(y−µ)2

2σ2 y ∈ Rn

Product of univariate Gaussian densities, for i.i.d.Gaussians

f (y1, . . . , yn) =∏

i

1√2πσ

e−(yi−µ)2

2σ2

=1

(2π)n/2σn e−P

i(yi−µ)2

2σ2 y1, . . . , yn ∈ R

The general multivariate Gaussian case

f (x) =1

(2π)n/2|Σ|12

e−12 (x−µ)′Σ−1(x−µ) x ∈ Rn

(is a density on Rn if Σ > 0 and µ ∈ Rn)

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Linear combinations of Gaussian randomvariables

Linear combinations (aX + bY , c′Y etc) of Gaussianrandom variables (independent or jointly multivariateGaussian) are also Gaussian.E [aX + bY ] = aµX + bµYV [aX + bY ] = a2σ2

X + b2σ2Y if independent

V [aX + bY ] = a2σ2X + b2σ2

Y + 2abCov(X ,Y ) in generalE [c′Y] = c′µV [c′Y] = c′ΣY cV [AY] = AΣY A′

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Distributions of estimatesThe number b should be viewed as the outcome of therandom variable,

β =

∑i(xi − x)Yi∑i(xi − x)2

(note the rewrite from earlier formula). So β is a linearcombination of Gaussian Y1, . . . ,Yn so β is Gaussian.Will show that E [β] = β, find V [β] = σ2

β= . . . and

β − βσ2β

∼ tn−2

Inference: Hypothesis testing and confidence intervals.Example: Test whether slope is zero (H0 : β = 0) i.e.whether there is some relationship between x and y .Never be content with a mathematical or conceptualmodel: Insist that it is justified by data!

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Outline

The course



Other things

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Goodness of fit and diagnostics: STATS 310xxxVerifying SLR assumptions...Will derive tests for nonlin-earity (lack-of-fit), normal-ity, homoscedasticity, in-dependence, outliers, etcetc.This is (mainly) based onresiduals ei = yi − yi orvariations thereofSome concepts: Stan-dardized residuals,studentized residuals,deleted residuals etc etc.

●

●●●●●●

●

●●

●●●

●●●●

●●

●●

●

●●

●●●

●●●●●

●●

●●

●

●

●●

●●

●

●●

●

●●●

●

0 10 20 30 40 5020

4060

80

x

y

●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

0 10 20 30 40 50

−2

02

4

x

resi

d(fm

)

Simplest diagnostics: Plot residuals in all possi-

ble ways

Note: It is never enough to fit a model or use it forpredictions. One must always also verify whether themodel is adequate.

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Outline

The course



Other things

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

SLR in matrix form

y ∈ Rn = vector of measurements

X =

1 x1...

...1 xn

the “X-matrix”

min∑

(yi − (α+ βxi))2 is equiv-alent to finding

β =

(αβ

)to mininmize ||y− Xβ||2Number notation: y = Xβ + e

Point estimate as a projection

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Multiple regression: STATS 310 mulreg

The model:y = Xβ + e

where X in an n × p matrix.Example: Interaction model: yi = α + βxi + γwi + δxiwi .Defining xi1 = 1, xi2 = xi , xi3 = wi , xi4 = xiwi , thisbecomes a multiple linear regression model.More examples: Estimate single intercept, many slopes;Test whether multiple lines are all parallel; ...Used in all fields of biology (fisheries, genetics, ...),economics, psychology, sociology, ...Need to develop point estimates, methods of validationand testing.

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Matrix solution

The point estimate is

β = (X′X)−1X′Y

which is always unbiased (if the mean of the Y -s iscorrect)

E[β]

= β

and has variance-covariance matrix

V[β]

= σ2(X′X)−1.

(if the variance assumptions are correct)and is multivariate Gaussian (if the Y -values areGaussian).

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Outline

The course



Other things

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

TestingStatistical tests (of modelreductions) can use pro-jections in Rn where o.n.bases give SSEs and d.f.

y = ζ1u1 + . . . ζquq

+ζq+1uq+1 + . . . ζr ur

+ζr+1ur+1 + . . . ζnun

SSE(F ) = ||y− Xβ||2 =n∑

i=p+1

ζ2i

SSE(F )− SSE(R) = ||Zγ − Xβ||2 =

p∑i=r+1

ζ2i

SSE(R) = ||y− Zγ||2 =n∑

i=r+1

ζ2i

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Estimable functions: STATS 310 xxx

In the one-way layout not all parameter combinations canbe estimated. Those linear combinations of parameters,ψ = c′β which have an unbiased estimate using a linearcombination of y -valules are termed estimable functions.

y︸︷︷︸n×1

∼ n( X︸︷︷︸n×p

β︸︷︷︸p×1

, σ2 I︸︷︷︸n×n

)

ψi = c′iβ; C = (c1, . . . , cq)′; ψ = Cβ

ψ = Ay = Cβ ∼ n(Cβ, σ2AA′)

Theorem: ψ ∼ n(ψ,Σψ

), ||y−Xβ||2

σ2 ∼ χ2n−r and these

two quantities are independent.

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Outline

The course



Other things

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Model selection

Many x-variables?Need to choose subset for inclusionLook at all subsets?How should quality of fit be measured?Forward and backwards stepwise regression.Example: What drives recruitment to fish stocks? Have aseries of e.g. 50 years of data, but several dozens ofpossible x-values.

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Model validation

Validate mainly using var-ious residualsInvestigate assumptions,but now also investigateinfluential observationsDFFITS etcHat matrix H, particularlythe leverage values hii

H = X(X′X

)−1 X′

projects y to yInvestigate collinearity

●

●

●●●●

●●

●

●

●

0 10 20 30 40 50

−2

02

46

810

●

●

●

●● ●

●

●

●

●

●

●

●

● ●

●

●●

●

●

5 10 15 20 25

−1

01

23

45

6

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●●

●●

●●●

●

●●●●●

●●●

●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●

●●

●

●●●●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●●

●●

●

●

0 2 4 6 8 10

02

46

8

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●

●

●●

●

●

●

●

●

●

●●

●●●

●

●

●

●●●●

●

●

●

●●

●●●

●

●●●●●●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●●

●●●

●●

●

●

●

●

●

●

●●●

●

●

●

●●●

●

●

●

●

●●●●

●●●●

●

●●

●

●●●●

●

●

●

●

●●

●●●

●●●●

●

●●

●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●

●●●

●

●

●

●

●

●●

●

●●

●

●

●●

●

●

●●

●

●●●

●

●●

●

●●

0 50 100 200 300

−2

02

46

8

Examples of problem cases

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Analysis of variance: STATS 310 anova

One-way layout: yij = µ+ αi + eijThis is a particular linear model (multiple regressionmodel), but with special properties! Will developexpressions for solutions, sums of squares etc etc.Example: Breast feeding and IQGroup n IQbar sI 90 92.8 15.2 not breast fedIIa 17 94.8 19.0 failed breast feedingIIb 193 103.7 15.3 breast fed

More examples: Fertilizers, crops; differential geneexpressions; ...Two-way layout etc etc

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Multiple comparisons procedures: ScheffeTheorem: Under the above assumptions and definitions,(

ψ −ψ)′

B−1(ψ −ψ

)/q

||y− Xβ||2/(n − r)∼ Fq,n−r

Pβ

[(ψ −ψ

)′B−1

(ψ −ψ

)≤ qs2Fq,n−r ,1−α

]= 1− α

If Ψ = {ψ = k1ψ1 + . . .+ kqψq}, then

P[ψ −

√qF ∗σψ ≤ ψ ≤ ψ +

√qF ∗σψ ∀ψ ∈ Ψ

]= 1− α

and we are therefore allowed to search among allestimable functions within the set to find significanteffects.Can now do legal data-snooping!

ALSM

GS

The course


Some theory

Model evaluation


Some more theory

Applied stuff

Other things

Generalizations and other topics

I The Generalized linear model (GLM) drops theassumption of normality and defines a “link function”

I The Generalized Additive Model (GAM) allowssmoothing function in place of linear combinations.

I Random effects modelsI Nonlinear modelsI Correlated observations

Applied linear statistical models: An overviewgunnar/kennsla/alsm/alsmintro/al...Applied linear...

Documents

Transcript of Applied linear statistical models: An overviewgunnar/kennsla/alsm/alsmintro/al...Applied linear...