2013 06 19 Robust Regression

8/13/2019 2013 06 19 Robust Regression

1/101


2/101


3/101

Introduction

Introduction

Workshop Modern methods for robust regression, (# 152, Robert Andersen) 3/45

I d i


4/101

Introduction

Modern regression

(OLS Ordinary Least Squares) regression:One of the most widely used statistical methods in socialsciences.

However, we will see that OLS regression does have limitations.

Modern regression methods can be seen as improvements/alternatives to the usual regression model.

Robust regression is one such modern method.


I t d ti


5/101

Introduction

Robust regression

The termrobusthas multiple interpretations in estimationframeworks:


Introduction


6/101

Introduction

Robust regression


Robustness of validity: Resistance of the estimator to unusualobservations.A robustestimator should not suffer big changes when small changes are

made to the data.


Introduction


7/101

Introduction

Robust regression



made to the data.

Robustness of efficiency: Resistance of the estimator toviolations of underlying distributional assumptions.A robustestimator should keep high precision (i.e., small SEs) when

distributional assumptions are violated.


Introduction


8/101

Introduction

Robust regression



made to the data.

Robustness of efficiency: Resistance of the estimator toviolations of underlying distributional assumptions.A robustestimator should keep high precision (i.e., small SEs) when

distributional assumptions are violated.

We will mostly focus onrobustness of validityin regression.


Introduction


9/101

Introduction

Limitations of OLS regression: Example

Jasso, G. (1985). Marital coital frequency and the passage of time: Estimating

the separate effects of spouses ages and marital duration, birth and marriagecohorts, and period influences. American Sociological Review, 50(2), 224-41.doi:10.2307/2095411

Jasso (1985) found that wifes age had apositive effecton the

monthly coital frequency of married couples, after controlling forperiod and cohort effects.


Introduction


10/101

Introduction






Kahn and Udry (1986) questioned her findings:

They found four miscodes in the dataset. They found four additional outliersusing model diagnostics. They claim Jasso failed to consider a relevant interaction effect

(length of marriage by wifes age): Model misspecification.


Introduction


11/101






Kahn and Udry (1986) questioned her findings:

They found four miscodes in the dataset. They found four additional outliersusing model diagnostics. They claim Jasso failed to consider a relevant interaction effect

(length of marriage by wifes age): Model misspecification.

Total sample size: 2062.


Introduction


12/101


Model 1 Model 2Period 0.72 0.67

Log Wifes Age 27.61 13.56Log Husbands Age 6.43 7.87Log Marital Duration 1.50 1.56

Wife Pregnant 3.71

3.74

Child Under 6 0.56 0.68

Wife Employed 0.37 0.23Husband Employed 1.28 1.10

R2 .0475 .0612

n 2062 2054

Note. p< .10, p< .05, p< .01.

Model 1: Jassos (1985) original model.Model 2: Kahn and Udrys (1986) model excluding 4 miscodes and 4

outliers.Workshop Modern methods for robust regression, (# 152, Robert Andersen) 7/45

Introduction


13/101


Some conclusions:

A small number of unusual observations can have a large effecton the estimated regression coefficients.



14/101

Introduction


15/101


Some conclusions:


Large samples arenotimmune to this problem.(Previous example: 8/2062= 0.39% of data.)

Using diagnostic tools to uncover potential problems is a crucial(and often disregarded) analysis step.


Introduction


16/101


Some conclusions:


Large samples arenotimmune to this problem.(Previous example: 8/2062= 0.39% of data.)

Using diagnostic tools to uncover potential problems is a crucial(and often disregarded) analysis step.

The decision on what to do when influential observations arefound should be based onsubstantive knowledge(i.e., no one-way-out solution exists).



17/101

Important background


18/101

Properties of estimators

Assessing whether an estimator is robustrequires checking severalmathematical properties.

Notation:

population parameter that we intend to estimate

T estimator for Y sample (Y = (y1, . . . , yn))

estimate of (T(Y) =

)

n sample size




19/101


1 Bias: Does the estimator give, on average, the desiredparameter?

bias=E[ ]



20/101


21/101



22/101


4 Influence function(IF): Localmeasure of the resistance of anestimator.Ideal: Having aboundedinfluence function, implying that onesingle observation has limited influence on the estimator.




23/101


4 Influence function(IF): Localmeasure of the resistance of anestimator.Ideal: Having aboundedinfluence function, implying that onesingle observation has limited influence on the estimator.

5 Relative efficiency: Ratio ofMSEs of the estimator withsmallest MSE (say TOpt) and estimator T

Relative efficiency=

MSE(TOpt)

MSE(T)

0 Rel. Effic. 1, the larger the better.




24/101


Fact:

OLS estimators are the most efficient (i.e., with smallestMSE) among all unbiased linear estimators if all theregression model assumption are met.




25/101


Fact:

OLS estimators are the most efficient (i.e., with smallestMSE) among all unbiased linear estimators if all theregression model assumption are met.

As a consequence:Relative efficiency(more precisely, asymptoticefficiency) ofrobust estimators is assessed in comparison to the OLSestimators when model assumptions are met.



26/101



27/101

Measures of location

Measure of location: Quantity that characterizes a position in a

distribution

Statistic Formula BDP IF In current use in

robust regression

Mean y=

i

yi

n 0 Unbounded No

-trimmed mean yt= y(g+1)++y(ng)

n2g Bounded Yes

Median M=Q.50 .50 Bounded Yes




28/101

Measures of location: Influence function



29/101


30/101

Robustness, resistance, and OLS regression

Cl ifi i f l b i


31/101

Classification of unusual observations

Observation Definition

Does it affect

regression estimates?



Cl ifi i f l b i


32/101



Does it affect


Univariate outlier Outlier ofx ory

Not necessarily(unconditional on each other)



Cl ifi ti f l b ti


33/101



Does it affect




Regressionoutlier Unusual yvalue for a given x Not necessarily (A)



Cl ifi ti f l b ti


34/101



Does it affect





Observation withleverage Unusual x value Not necessarily (B)



Cl ssifi ti f s l bs ti s


35/101



Does it affect





Observation withleverage Unusual x value Not necessarily (B)

Observation withinfluence Regression estimates change

Yes (C)greatly with/without them



36/101


37/101




38/101


Conclusion:

Being x-discrepant (leverage) ory-discrepant (regressionoutlier)is not sufficient to identifyinfluentialobservations.

However, it is a combination of both x- and y- discrepanciesthat determines the influence of an observation.





39/101


Conclusion:



Important note:Observations with high leverages that follow the main regressiontrend helpdecreasingthe SE of the estimates! Plot B

SE(b) = se

i(xi x)2





40/101


Conclusion:



Important note:Observations with high leverages that follow the main regressiontrend helpdecreasingthe SE of the estimates! Plot B

SE(b) = se

i(xi x)2

How do we identifyregressionoutliers, observations withleverage,and observations withinfluence?



Identifying unusual observations


41/101


n = number of observations (i=1, . . . , n)

k= number of predictors (j=1, . . . , k)

Detecting. . . Use. . . Rule of thumb?

Test? Plot?Flag if...





42/101






Leverage Hat values hi >2(k+ 1)/n a Index plot

a. For large samples. Use hi >3(k+1)/n for small samples.



43/101




44/101






Leverage Hat values hi >2(k+ 1)/n a Index plot

Regression outliers Studentized residuals | | 2 Yesb QQ-plot

InfluenceDFBETAs | | 2/n Index plotCooks D

> .5 c Index plot

>4/(n k 1)d

a. For large samples. Use hi >3(k+1)/n for small samples.b. Controlling for capitalization by chance is required.c. According to Cook and Weisberg (1999).d. According to Fox (1997).



Identifying unusual observations: Example


45/101


Weakliem, D.L., Andersen, R., & Heath, A.F. (2005). By popular demand:

The effect of public opinion on income quality. Comparative Sociology, 4(3),261-284. doi:10.1163/156913305775010124





46/101




We will focus on a subset of the original data: Sample: n=26 countries with democracies


47/101

y g p





48/101

y g p





49/101


50/101


51/101




52/101



53/101




54/101



55/101


56/101


57/101




58/101

Conclusion:

Slovakia and Czech Republic seem to have a large influence onthe estimation of the regression coefficients. This can be confirmed:

OLS OLS(all countries) (omitting cz and sk)

b SE b SE

Intercept .028 .128 .107* .058Gini .00074 .0028 .00527*** .0013GDP .0175** .0079 .0063 .0037

s .138 .0602

R2

.175 .4622n 26 24

Note. p< .10, p< .05, p< .01.

Lets now look into better alternatives to OLS!


Robust regression for the linear model


59/101

Robust regression for the

linear model



60/101


61/101


62/101


Various robust regression models


63/101

Type Estimator Breakdown Bounded Asymptotic

Point Infl. Func. Efficiency

OLS 0 No 100%

L-EstimatorsLAV (Least Absolute Values) 0 Yes 64%LMS (Least Median of Squares) .5 Yes 37%

LTS (Least Trimmed Squares) .5 Yes 8%

R-Estimators Bounded influence estimator < .2 Yes 90%

M-EstimatorsM-estimates (Huber, biweight) 0 No 95%GM-estimates (Mal.& Schw.) 1/(p+1) Yes 95%GM-estimates (S1S) .5 Yes 95%



64/101


Various robust regression models


65/101

Type Estimator Breakdown Bounded Asymptotic

Point Infl. Func. Efficiency

OLS 0 No 100%

L-EstimatorsLAV (Least Absolute Values) 0 Yes 64%LMS (Least Median of Squares) .5 Yes 37%

LTS (Least Trimmed Squares) .5 Yes 8%

R-Estimators Bounded influence estimator < .2 Yes 90%

M-EstimatorsM-estimates (Huber, biweight) 0 No 95%GM-estimates (Mal.& Schw.) 1/(p+1) Yes 95%GM-estimates (S1S) .5 Yes 95%

S-Estimators S-estimates .5 Yes 33%

GS-estimates .5 Yes 67%

MM-Estimators MM-estimates .5 Yes 95%



66/101


67/101


M-Estimators


68/101

Objectivefunctions:

Weightfunctions:



69/101


GM-, S-, GS-, MM-estimators


70/101

GM-Estimators: Generalized M-estimators that take both vertical

outliers and leverage into account: Vertical outliers: By using M-estimators.

Leverage: Using hat values.

Method to retain: Schweppe one-step estimator (S1S).

S-Estimators: Minimize a robust M-estimate of the residual scale(instead ofY scale).

GS-Estimators: Generalized S-estimates that increase efficiency.

MM-Estimators: Rely on both M- (high efficiency) and S- (highBDP) estimators.



Example: Simulated data


71/101


72/101




73/101




74/101



75/101


Using robust regression as regression diagnostics


76/101

Common statistics measuring influence of observations (e.g., Cooks

distance) are not robust against unusual obervations.



77/101


78/101


79/101



E l I d l f b i i h


80/101

Example: Index plots of robust regression weights wi

Idea: Weights indicate levels of unusualness (y- and/or x-discrepancies) the smaller, the more unusual.



81/101



E l RR l t (T k 1991)


82/101

Example: RR-plots (Tukey, 1991)

Idea: Robust regression residuals are better than OLS residuals fordiagnosing outliers:

OLS regression tries to produce normal-looking residuals

even when the data themselves are not normal.(Rousseeuw & van Zomeren, 1990)

Workshop Modern methods for robust regression, (# 152, Robert Andersen) 36/45 Robust regression for the linear model


E a le RR lots (T ke 1991)


83/101

Example: RR-plots (Tukey, 1991)

Idea: Robust regression residuals are better than OLS residuals fordiagnosing outliers:

OLS regression tries to produce normal-looking residuals

even when the data themselves are not normal.(Rousseeuw & van Zomeren, 1990)

RR-plots: Residual-residual scatterplot matrix

OLS assumptions hold = scatter around y=x line(OLS vsrobust regression residuals)

Workshop Modern methods for robust regression (# 152 Robert Andersen) 36/45 Robust regression for the linear model



84/101

Workshop Modern methods for robust regression (# 152 Robert Andersen) 37/45


85/101

Standard errors for robust regression


Analytical (asymptotic) SEs are available for only some types of


86/101

Analytical (asymptotic) SEs are available for only some types ofrobust regression: M- and GM-estimation. S- and GS-estimation. MM-estimation.

These estimates are given by

SE = Diagonalvar-cov matrix.

Workshop Modern methods for robust regression (# 152 Robert Andersen) 39/45 Standard errors for robust regression


Analytical (asymptotic) SEs are available for only some types of


87/101



SE = Diagonalvar-cov matrix. Some problems with SE: Unreliable for small n (say,


88/101



SE = Diagonalvar-cov matrix. Some problems with SE: Unreliable for small n (say,


89/101

Bootstrapping is useful to estimate difficult/unknown sampling

distributions.


Computing standard errors: Bootstrapping



90/101


distributions.Remember: If OLS assumptions are met, then OLS estimates for theSEs are better than bootstrap estimates!



91/101


92/101


93/101


Computing bootstrapping CIs

Having estimated regression coefficients plustheir associated


94/101

g g p

(bootstrapped) SEs, it is now possible to compute (1

)% CI.





95/101

(bootstrapped) SEs, it is now possible to compute (1 )% CI. There are some possibilities:

Bootstrap t CI: When bias is small and the bootstrap samplingdistribution is roughly normally distributed.

CI= tnk1,/2SE()



96/101





97/101

(bootstrapped) SEs, it is now possible to compute (1 )% CI. There are some possibilities:

Bootstrap t CI: When bias is small and the bootstrap samplingdistribution is roughly normally distributed.

CI= tnk1,/2SE() Bootstrap percentile CI: When bias is small but the bootstrap

sampling distribution deviates from the normal distribution.

CI= (q/2(), q1/2()), =bootstrapped dist.

Bias-corrected percentile CI: When bias is large (e.g., for smallsample sizes).

W ksh M d th ds f b st ssi (# 152 R b t A d s ) 42/45


98/101


Example: Public opinion about pay inequality


99/101

Bootstrap t CIs

B Boot. SE Lower 95% Upper 95%

Intercept .0978 .0580Gini .0051 .0012 .0026 .0076GDP .0057 .0033 .0015 .0129

W k h M d th d f b t i (# 152 R b t A d ) 43/45 Robust regression in R


100/101

Robust regression in R

W k h M d th d f b t i (# 152 R b t A d ) 44/45 Robust regression in R

Robust regression in R

Fitting regression models


101/101

Fitting regression models

Reg. Model R package R command

OLS lm(SECPAY gini + GDP)L-Estimation (LAV) quantreg rq(SECPAY gini + GDP)M-Estimation (Huber) MASS rlm(SECPAY gini + GDP)M-Estimation (Biweight) MASS rlm(SECPAY gini + GDP,psi=psi.bisquare)MM-Estimation MASS rlm(SECPAY

gini + GDP,method="MM")

Model diagnostics (stats package, loaded by default)

Statistic Diagnosing. . . R command

Hat values Leverage hatvalues(lm(SECPAY

gini + GDP))

Studentized residual Reg. outlier rstudent(lm(SECPAY gini + GDP))DFBETA Influence dfbeta(lm(SECPAY gini + GDP))Cooks D Influence cooks.distance(lm(SECPAY gini + GDP))

W k h M d h d f b i (# 152 R b A d ) 45/45

2013 06 19 Robust Regression

Documents

Transcript of 2013 06 19 Robust Regression