Simple and multiple regression analysis in matrix form
-
Upload
jerome-grimes -
Category
Documents
-
view
105 -
download
1
description
Transcript of Simple and multiple regression analysis in matrix form
Simple and multiple regression analysis in matrix form Least square Beta estimation Simple linear regression Multiple regression with two predictors Multiple regression with three predictors Sum of square R2
Test on parameters Covariance matrix of the Standard error of the
Simple and multiple regression analysis in matrix form Tests on individual predictors Variance of individual predictors Correlation between predictors Standardized matrices Correlation matrices Sum of squares in Z R2 in Z
R2 between independent variables Standard error of in Z
Least squareStarting from the general:
The method of least squares estimate of the beta parameter minimizing the sum of squares due to error.
In fact, if:
You can estimate:
Least square
Simple linear regression
Simple linear regression
Simple linear regression
Simple linear regression
intercepts
slope
Multiple regression Similar to the simple A single dependent variable (Y) Two or more independent
variables (X) Multiple correlation (rather than
simple) Estimation by least squares
Simple linear regression (var.: 1 dep., 1 indep.)
Multiple linear regression (Var.:1 dep., 2 indep.)
intercepts errorIndependent variables
slope
Multiple regression
Multiple regression matrix form
Multiple regression matrix form
Multiple regression matrix form
X’X
inversa
Multiple regression matrix form
Multiple regression matrix form
In matrix notation is briefly expressed :
Multiple regression with three predictors
Multiple regression with three predictors
Matrix form
Matrix form
Matrix form
Matrix form
General scheme
General scheme
The least squares method allows to check the following equality:
Sum squares
Since in general:it's possible to derive that the sum of the squares of the distances of y from its average can be decomposed into the sum of squares due to regression and the sum of squares due to error, according to:
Sum squares
Sum squares
It should be noted the equivalence of :
Sum squares
Sum squares
In summary :
Sum squares
R2
Adjusted R2YY’
Because the coefficient of determination depends on both the number of observations (n) that the number of independent variables (k) it is convenient to correct by the degrees of freedom.
Adjusted R2YY’
In our example :
Once a regression model has been constructed, it may be important to confirm the goodness of fit (R-squared )of the model and the statistical significance of the estimated parameters. Statistical significance can be checked by an F-test of the overall fit, followed by t-tests of individual parameters
Test on parameters
You can test the hypothesis of differences with 0 of the parameters i taken together :
Test on parameters
k= Number of columns of the matrix X excluding X0
n= Number of observations in y
Test on parameters
Test on parameters
k= Number of columns of the matrix X excluding X0
n= Number of observations in y
Covariance matrix of the
An estimate of the covariance matrix of the beta values result by:
We denote:
Covariance matrix of the
Where the diagonal elements are an estimate of the variance of the single i
Standard error of the
The standard error of the parameters can be calculated with the following formula:
where cii is the diagonal element inside the matrix(X’X)-1 corresponding to the parameter i .
Standard error of the
Nota: quando il valore di cii è elevato il valore di sebi
cresce, indicando che la variabile Xi ha un alto coefficiente di correlazione multipla con le altre variabili X.
Standard error of the
the increase in R2i led to a decreases of the
denominator of the ratio and, consequently, increases the value of the standard error of the parameter i.
The standard error of the i can also be calculated in the following way:
where
With the standard error of measurement associated with each i you can make a t-test to verify:
Tests on individual predictors
Tests on individual predictors
With the standard error of measurement associated with each i is also possible to estimate the confidence interval for each parameter:
Tests on individual predictors
1. Calculate the SSreg for the model containing all the independent variables.
2. Calculate the SSreg for the model excluding the variable for which you want to test the significance (SS-i).
3. Perform an F-test with the numerator equal to the difference SSreg-SSi weighted for the difference between the degrees of freedom of the two models, and with denominator SSREs / (nk-1).
In order to conduct a statistical test on the regression coefficients is necessary:
Tests on individual predictors
To test, for example, only the weight of the first predictor compared to the total model, it is necessary to calculate a new matrix i from the matrix Xi which was taken off the column belonging to the first predictor. From this follows immediately the calculation of SSi.
Tests on individual predictors
Tests on individual predictors
Same procedure is followed to test any subset of predictors.
Similarly we have:
Tests on individual predictors
It is interesting to note that this test on a single predictor is equivalent to the t-test b1 = 0. When the numerator there is only one degree of freedom, that is in fact the equivalence:
Summary table
On this occasion, none of the estimated parameters obtained statistical significance on the hypothesis i 0
Variance of individual predictors Xi
Using the matrix X'X we can calculate the variance of each variable Xi .
Variance of individual predictors Xi
Covariance between predictors and the dependent variable
It is possible to calculate the covariance between the independent variables and the dependent variable according to:
Covariance between predictors and the dependent variable
The correlation between the independent variables and the dependent variable is given by:
As we will see later the use of standardized matrices simplifies the calculation immediately.
Test on multiple predictor You can perform a statistical test on a group
of predictors in order to verify the significance.
To do this, you use the formula specified above :
To test, for example, the weight of only the first and second predictors with respect to the total model, it is necessary to calculate a new matrix i from the matrix Xi which was taken off the column belonging to these predictors. From this follows immediately the calculation of SSi.
Test on multiple predictor
Correlation between predictors
Standard condition of independence between the variables Xi
Correlation between predictors
Condition of dependence between variables Xi
Completely standardized solution.
We denote by Ri. the multiple correlation of the variable Xi with the remaining variables, denoted by Xj
Correlation between predictors
The element cii represents the value of the diagonal of the matrix (X'X) -1 while S2
i is the variance of the variable Xi.
In case you do not have the X'X matrix but you have the MSres and the standard error of the parameter i, the correlation between one X and the other one can be calculated in the following manner:
Correlation between predictors
Correlation between predictors
The X matrix and the y matrix can be converted into standardized scores by dividing the deviation of each element from the average for the appropriate standard deviation.
Standardized matrices
In our example we have:
Standardized matrices
Standardized matrices
With standardized variables is not necessary to include in the matrix Z the component 1 as the parameter 0 is equal to 0.
The standardized coefficients can be obtained from those non-standardized using the formula:
The equation of the regression line becomes:
Standardized matrices
Standardized matrices
In our example we have:
Use standardized matrices allows to set the parameter 0 = 0. In fact, if the variables are standardized the intercept value for Y is 0, since all the means are equal to 0;Inoltre, essendo
the correlation between any two standardized variables is:
with i, j between 1 and k.
Standardized matrices
Correlation matrices
If we multiply the matrix (Z'Z) for the scalar [1 / (n-1)] we obtain the correlation matrix R between the independent variables
In our example we have:
Correlation matrices
Correlation of Y with individual predictors
Similarly if the variable Y is also standardized and multiply the product by the scalar Z'Yz [1 / (n-1)] we obtain the correlation matrix ryi of the variable Y with its predictors Xi.
Correlation of Y with individual predictors
The solution of the system of normal equations of the line leads to the following equation:
The estimated values can be obtained using the equation:
Correlation of Y with individual predictors
With standardized variables we have:
Starting from the general formulas it's possible to have the following simplified formulas:
Sum of squares in Z
Sum of squares in Z
Calculation of R2y.123
Having decomposed the variance component due to the regression and the component due to the residuals, it is immediate to calculate:
Multiple correlation between the Xi.yz
If in general the squared multiple correlation of a variable with the other independent Xi is:
in the presence of standardized variables, it becomes:
where the element aii belongs to the diagonal of the matrix R-1.
If you want to calculate the other two coefficients now you will have to do the following:
For example, the squared multiple correlation between the first variable X1 and the other two can be calculated in the following way:
Multiple correlation between the Xi.yz
Standard error of z
The standard error of the standardized parameters is obtainable by the general formula:
Standard error of z
You now have all the elements to test the differences of individual predictors from 0, obtaining the same results obtained with the non-standardized variables.