Simple and multiple regression analysis in matrix form Least square Beta estimation Simple linear regression Multiple regression with two predictors Multiple regression with three predictors Sum of square R2

Test on parameters Covariance matrix of the Standard error of the

Simple and multiple regression analysis in matrix form Tests on individual predictors Variance of individual predictors Correlation between predictors Standardized matrices Correlation matrices Sum of squares in Z R2 in Z

R2 between independent variables Standard error of in Z

Least squareStarting from the general:

The method of least squares estimate of the beta parameter minimizing the sum of squares due to error.

In fact, if:

You can estimate:

Least square

Simple linear regression

Simple linear regression

Simple linear regression

Simple linear regression

intercepts

slope

Multiple regression Similar to the simple A single dependent variable (Y) Two or more independent

variables (X) Multiple correlation (rather than

simple) Estimation by least squares

Simple linear regression (var.: 1 dep., 1 indep.)

Multiple linear regression (Var.:1 dep., 2 indep.)

intercepts errorIndependent variables

slope

Multiple regression

Multiple regression matrix form

Multiple regression matrix form

Multiple regression matrix form

X’X

inversa

Multiple regression matrix form

Multiple regression matrix form

In matrix notation is briefly expressed :

Multiple regression with three predictors

Multiple regression with three predictors

Matrix form

Matrix form

Matrix form

Matrix form

General scheme

General scheme

The least squares method allows to check the following equality:

Sum squares

Since in general:it's possible to derive that the sum of the squares of the distances of y from its average can be decomposed into the sum of squares due to regression and the sum of squares due to error, according to:

Sum squares

Sum squares

It should be noted the equivalence of :

Sum squares

Sum squares

In summary :

Sum squares

R2

Adjusted R2YY’

Because the coefficient of determination depends on both the number of observations (n) that the number of independent variables (k) it is convenient to correct by the degrees of freedom.

Adjusted R2YY’

In our example :

Once a regression model has been constructed, it may be important to confirm the goodness of fit (R-squared )of the model and the statistical significance of the estimated parameters. Statistical significance can be checked by an F-test of the overall fit, followed by t-tests of individual parameters

Test on parameters

You can test the hypothesis of differences with 0 of the parameters i taken together :

Test on parameters

k= Number of columns of the matrix X excluding X0

n= Number of observations in y

Test on parameters

Test on parameters

k= Number of columns of the matrix X excluding X0

n= Number of observations in y

Covariance matrix of the

An estimate of the covariance matrix of the beta values result by:

We denote:

Covariance matrix of the

Where the diagonal elements are an estimate of the variance of the single i

Standard error of the

The standard error of the parameters can be calculated with the following formula:

where cii is the diagonal element inside the matrix(X’X)-1 corresponding to the parameter i .

Standard error of the

Nota: quando il valore di cii è elevato il valore di sebi

cresce, indicando che la variabile Xi ha un alto coefficiente di correlazione multipla con le altre variabili X.

Standard error of the

the increase in R2i led to a decreases of the

denominator of the ratio and, consequently, increases the value of the standard error of the parameter i.

The standard error of the i can also be calculated in the following way:

where

With the standard error of measurement associated with each i you can make a t-test to verify:

Tests on individual predictors

Tests on individual predictors

With the standard error of measurement associated with each i is also possible to estimate the confidence interval for each parameter:

Tests on individual predictors

1. Calculate the SSreg for the model containing all the independent variables.

2. Calculate the SSreg for the model excluding the variable for which you want to test the significance (SS-i).

3. Perform an F-test with the numerator equal to the difference SSreg-SSi weighted for the difference between the degrees of freedom of the two models, and with denominator SSREs / (nk-1).

In order to conduct a statistical test on the regression coefficients is necessary:

Tests on individual predictors

To test, for example, only the weight of the first predictor compared to the total model, it is necessary to calculate a new matrix i from the matrix Xi which was taken off the column belonging to the first predictor. From this follows immediately the calculation of SSi.

Tests on individual predictors

Tests on individual predictors

Same procedure is followed to test any subset of predictors.

Similarly we have:

Tests on individual predictors

It is interesting to note that this test on a single predictor is equivalent to the t-test b1 = 0. When the numerator there is only one degree of freedom, that is in fact the equivalence:

Summary table

On this occasion, none of the estimated parameters obtained statistical significance on the hypothesis i 0

Variance of individual predictors Xi

Using the matrix X'X we can calculate the variance of each variable Xi .

Variance of individual predictors Xi

Covariance between predictors and the dependent variable

It is possible to calculate the covariance between the independent variables and the dependent variable according to:

Covariance between predictors and the dependent variable

The correlation between the independent variables and the dependent variable is given by:

As we will see later the use of standardized matrices simplifies the calculation immediately.

Test on multiple predictor You can perform a statistical test on a group

of predictors in order to verify the significance.

To do this, you use the formula specified above :

To test, for example, the weight of only the first and second predictors with respect to the total model, it is necessary to calculate a new matrix i from the matrix Xi which was taken off the column belonging to these predictors. From this follows immediately the calculation of SSi.

Test on multiple predictor

Correlation between predictors

Standard condition of independence between the variables Xi

Correlation between predictors

Condition of dependence between variables Xi

Completely standardized solution.

We denote by Ri. the multiple correlation of the variable Xi with the remaining variables, denoted by Xj

Correlation between predictors

The element cii represents the value of the diagonal of the matrix (X'X) -1 while S2

i is the variance of the variable Xi.

In case you do not have the X'X matrix but you have the MSres and the standard error of the parameter i, the correlation between one X and the other one can be calculated in the following manner:

Correlation between predictors

Correlation between predictors

The X matrix and the y matrix can be converted into standardized scores by dividing the deviation of each element from the average for the appropriate standard deviation.

Standardized matrices

In our example we have:

Standardized matrices

Standardized matrices

With standardized variables is not necessary to include in the matrix Z the component 1 as the parameter 0 is equal to 0.

The standardized coefficients can be obtained from those non-standardized using the formula:

The equation of the regression line becomes:

Standardized matrices

Standardized matrices

In our example we have:

Use standardized matrices allows to set the parameter 0 = 0. In fact, if the variables are standardized the intercept value for Y is 0, since all the means are equal to 0;Inoltre, essendo

the correlation between any two standardized variables is:

with i, j between 1 and k.

Standardized matrices

Correlation matrices

If we multiply the matrix (Z'Z) for the scalar [1 / (n-1)] we obtain the correlation matrix R between the independent variables

In our example we have:

Correlation matrices

Correlation of Y with individual predictors

Similarly if the variable Y is also standardized and multiply the product by the scalar Z'Yz [1 / (n-1)] we obtain the correlation matrix ryi of the variable Y with its predictors Xi.

Correlation of Y with individual predictors

The solution of the system of normal equations of the line leads to the following equation:

The estimated values can be obtained using the equation:

Correlation of Y with individual predictors

With standardized variables we have:

Starting from the general formulas it's possible to have the following simplified formulas:

Sum of squares in Z

Sum of squares in Z

Calculation of R2y.123

Having decomposed the variance component due to the regression and the component due to the residuals, it is immediate to calculate:

Multiple correlation between the Xi.yz

If in general the squared multiple correlation of a variable with the other independent Xi is:

in the presence of standardized variables, it becomes:

where the element aii belongs to the diagonal of the matrix R-1.

If you want to calculate the other two coefficients now you will have to do the following:

For example, the squared multiple correlation between the first variable X1 and the other two can be calculated in the following way:

Multiple correlation between the Xi.yz

Standard error of z

The standard error of the standardized parameters is obtainable by the general formula:

Standard error of z

You now have all the elements to test the differences of individual predictors from 0, obtaining the same results obtained with the non-standardized variables.