Chapter 3 Simple Regression. What is in this Chapter? This chapter starts with a linear regression...

Chapter 3

Simple Regression

What is in this Chapter?

• This chapter starts with a linear regression model with one explanatory variable, and states the assumptions of this basic model

• It then discusses two methods of estimation: the method of moments (MM) and the method of least squares (LS).

• The method of maximum likelihood (ML) is discussed in the appendix

3.1 Introduction

Example 1: Simple Regression

y = sale

x = advertising expenditures

Here we try to determine the relationship between

sales and advertising expenditures.

3.1 Introduction

Example 2: Multiple Regression

y = consumption expenditures of a family

x1 = family income

x2 = financial assets of the family

x3 = family size

3.1 Introduction

• There are several objectives in studying these relationships.

• They can be used to:

1. Analyze the effects of policies that involve changing the individual x's. In Example 1 this involves analyzing the effect of changing advertising expenditures on sales

2. Forecast the value of y for a given set of x's.

3. Examine whether any of the x's have a significant effect on y.

3.1 Introduction• Given the way we have set up the problem until

now, the variable y and the x variables are not on the same footing

• Implicitly we have assumed that the x's are variables that influence y or are variables that we can control or change and y is the effect variable.

• There are several alternative terms used in the literature for y and x1, x2,..., xk.

• These are shown in Table 3.1.

3.1 Introduction

Table 3.1 Classification of variables in regression analysis

(a) Presdictand Predictors

(b) Regressand Regressors

(c) Explained variable Explanatory variables

(d) Dependent variable Independent variables

(e) Effect variable Causal variables

(f) Endogenous variable Exogenous variables

(g) Target variable Control variables

y x1, x2,……, xk

3.2 Specification of the Relationships

• As mentioned in Section 3.1, we will discuss th

e case of one explained (dependent) variable,

which we denote by y, and one explanatory (in

dependent) variable, which we denote by x.

• The relationship between y and x is denoted by

y = f(x)

Where f(x) is a function of x


• Going back to equation (3.1), we will assume that the function f(x) is linear in x, that is,

• And we will assume that this relationship is a stochastic relationship, that is,

Where ,which is called an error or disturbance, has a known probability distribution (i.e., is a random variable).

xxf )(

xy


• In equation (3.2), is the deterministic

component of y and u is the stochastic or

random component.

• and are called regression coefficients or

regression parameters that we estimate from the

data on y and x

x


• Why should we add an error term u ?

• What are the sources of the error term u in equati

on (3.2)?

• There are three main sources：


1. Unpredictable element of randomness in

human responses.

ex. If y =consumption expenditure of a household and

x = disposable income of the household, there is

an unpredictable element of randomness in each

household's consumption.

The household does not behave like a machine.


2. Effect of a large number of omitted variables.

• Again in our example x is not the only variable

influencing y. The family size, tastes of the family, spending habits, and so on, affect the variable y.

• The error u is a catchall for the effects of all these variables, some of which may not even be quantifiable, and some of which may not even be identifiable.


3. Measurement error in y. • In our example this refers to measurement error in the

household consumption. That is, we cannot measure it accurately.


• If we have n observations on y and x, we can

write equation (3.2) as

• Our objective is to get estimates of the unknown

parameters and in equation (3.3) given

the n observations on y and x.

)3.3(,....,2,1 niuxy iii


• To do this we have to make some assumption

about the error terms . The assumptions we

make are:

1. Zero mean.

2. Common variance.

3. Independence. and are independent for all iu

.allfor0)( iuE i

.allfor)var( 2 iui

ji ju

iu


4. Independence of . and are independent for all i and j. This assumption automatically follows if

are considered nonrandom variables. With reference to Figure 3.1, what this says is that the distribution of u does not depend on the value of x.

5. Normality, are normally distributed for all i. In conjunction with assumptions 1, 2, and 3 this implies that are independently and normally distributed with mean zero and a common variance . We write this as ),0(NI~ 2iu

jx jxiu

jx

iu

iu

2


• These are the assumptions with which we start.

We will, however, relax some of these

assumptions in later chapters.

• Assumption 2 is relaxed in Chapter 5.




• We will discuss three methods for estimating the

parameters and :

• 1. The method of moments (MM).

• 2. The method of least squares (LS).

• 3. The method of maximum likelihood (ML).

3.3 The Method of Moments

• The assumptions we have made about the error term u imply that

• In the method of moments, we replace these conditions by their sample counterparts.

• Let and be the estimators for and , respectively. The sample counterpart of is the estimated error (which is also called the residual), defined as

0),(covand0)( uxuE

iu

iuiii xyu ˆˆˆ


• The two equations to determine and are obtained by replacing population assumptions by their sample counterparts:


• In these and the following equations, denotes

. Thus we get the two equations

• These equations can be written as (noting that )

ni 1

0)xˆ-ˆ-y(or0ˆ ii iu

0)xˆ-ˆ-y(xor0ˆ iii ii ux

ˆˆ n

2ˆˆ

ˆˆ

iiii

ii

xxyx

xny

3.4 The Method of Least Squares

• The method of least squares requires that we should choose and as estimates of and , respectively, so that is a minimum.

• Q is also the sum of squares of the (within-sample) prediction errors when we predict given and the estimated regression equation.

• We will show in the appendix to this chapter that the least squares estimators have desirable optimal properties.

2

1

)ˆˆ(

n

iii xyQ

iy

ix


or

or (3.6)and

(3.7)

0)1)(ˆˆ(20ˆ

ii xyQ

ii xny ˆˆ

xy ˆˆ

0))(ˆˆ(20ˆ

iii xxyQ

2ˆˆ iiii xxxy


• Let us define

and

yxnyxyyxxS

ynyyyS

iiiixy

iiyy

)()(

)( 222

222)( xnxxxS iixx

xyS

S

xx

xy ˆˆandˆ


• The residual sum of squares (to be denoted by RSS) is given by

xyxxyy

iiii

ii

ii

SSS

xxyyxxyy

xxyy

xy

ˆ2ˆ

)()(ˆ2)(ˆ)(

)(ˆ

)ˆˆ(RSS

2

222

2

2


• But .Hence we have

• is usually denoted by TSS (total sum of squares) and is usually denoted by ESS (explained sum of squares).

• Thus TSS = ESS + RSS (total) (explained) (residual)

xxxy SS

xyyyxx

xyyy SS

S

SS RSS

2

yyS

yxS


• The proportion of the total sum of squares explained is denoted by ,where is called the correlation coefficient.

• Thus and .If is high (close to 1), then x is a good “explanatory” variable for y.

• The term is called the correlation determination and must fall between zero and 1 for any given regression.

2yx yx

TSSESS2 yx TSSRSS1 2 yx

2yx

2yx


• If is close to zero, the variable x explains very little of the variation in y. If is close to 1, the variable x explains most of the variation in y.

• The coefficient of determination is given by

2yx

2yx

2yx

yy

yxyx S

S

ˆ

TSS

RSSTSS

TSS

ESS2

3.9 Alternative Functional Forms for Regression Equations

• For instance, for the data points depicted in Fig

ure 3.7(a), where y is increasing more slowly th

an x, a possible functional form is y = α +β logx.

• This is called a semi-log form, since it involves t

he logarithm of only one of the two variables x a

nd y.


• In this case, if we redefine a variable X = log x, th

e equation becomes y = α + βX.

• Thus we have a linear regression model with the

explained variable y and the explanatory variabl

e X = log x.


• For the data points depicted in Figure 3.7(b), where y is increasing faster than x, a possible functional from is . In this case we take logs of both sides and get another kind of semi-log specification:

• If we define Y= log y and , we have

which is in the form of a linear regression equation.

xAey

xAy loglog

xY

Alog


• An alternative model one can use is

• In this case taking logs of both sides, we get

• In this case can be interpreted as an elasticity. Hence this form is popular in econometric work. This is call a double-log specification since it involves logarithms of both x and y. Now define Y= log y, X= log x, and .We have

which is in the form of a linear regression equation. An illustrative example is given at the end of this section.

Axy

xAy logloglog

Alog

XY


• Some other functional forms that are useful when the data points are as shown in Figure 3.8 are

or

• In the first case we define X=1/x and in the second case we define .In both case the equation is linear in the variables after the transformation.

xX 1

xY

xY


• Some other nonlinearities can be handled by what is known as “search procedures.”

• For instance, suppose that we have the regress equation

• The estimates of , ,and are obtained by minimizing 2

i

i xy

ux

y

Chapter 3 Simple Regression. What is in this Chapter? This chapter starts with a linear regression...

Documents

Transcript of Chapter 3 Simple Regression. What is in this Chapter? This chapter starts with a linear regression...