ECON 626: Applied Microeconomics - Economics.ozier.com

69
ECON 626: Applied Microeconomics Lecture 8: Permutations and Bootstraps Professors: Pamela Jakiela and Owen Ozier

Transcript of ECON 626: Applied Microeconomics - Economics.ozier.com

Page 1: ECON 626: Applied Microeconomics - Economics.ozier.com

ECON 626: Applied Microeconomics

Lecture 8:

Permutations and Bootstraps

Professors: Pamela Jakiela and Owen Ozier

Page 2: ECON 626: Applied Microeconomics - Economics.ozier.com

Part I: Randomization Inference

Page 3: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference vs Confidence Intervals

• See Imbens and Rubin, Causal Inference, first chapters.

• 100 years ago, Fisher was after a “sharp null,” where Neyman andGosset (Student) were concerned with average effects.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 3

Page 4: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

How can we do hypothesis testing without asymptotic approximations?

Begin with idea of a sharp null: Y1i = Y0i ∀i . (Gerber and Green, p.62)

• If Y1i = Y0i ∀i , then if we observe either, we have seen both.

• All possible treatment arrangements would yield the same Y values.

• We could then calculate all possible treatment effect estimates underthe sharp null.

• The distribution of these possible treatment effects allows us tocompute p-values: The probability that, under the null, somethingthis large or larger would occur at random. (For the two sided test,“large” means in absolute value terms.)

• This extends naturally to the case where treatment assignments arerestricted in some way. Recall, for example, the Bruhn andMcKenzie (2009) discussion of the many different restrictions thatcan be used to yield balanced randomization.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 4

Page 5: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

How can we do hypothesis testing without asymptotic approximations?Begin with idea of a sharp null: Y1i = Y0i ∀i . (Gerber and Green, p.62)

• If Y1i = Y0i ∀i , then if we observe either, we have seen both.

• All possible treatment arrangements would yield the same Y values.

• We could then calculate all possible treatment effect estimates underthe sharp null.

• The distribution of these possible treatment effects allows us tocompute p-values: The probability that, under the null, somethingthis large or larger would occur at random. (For the two sided test,“large” means in absolute value terms.)

• This extends naturally to the case where treatment assignments arerestricted in some way. Recall, for example, the Bruhn andMcKenzie (2009) discussion of the many different restrictions thatcan be used to yield balanced randomization.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 4

Page 6: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

How can we do hypothesis testing without asymptotic approximations?Begin with idea of a sharp null: Y1i = Y0i ∀i . (Gerber and Green, p.62)

• If Y1i = Y0i ∀i , then if we observe either, we have seen both.

• All possible treatment arrangements would yield the same Y values.

• We could then calculate all possible treatment effect estimates underthe sharp null.

• The distribution of these possible treatment effects allows us tocompute p-values: The probability that, under the null, somethingthis large or larger would occur at random. (For the two sided test,“large” means in absolute value terms.)

• This extends naturally to the case where treatment assignments arerestricted in some way. Recall, for example, the Bruhn andMcKenzie (2009) discussion of the many different restrictions thatcan be used to yield balanced randomization.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 4

Page 7: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

How can we do hypothesis testing without asymptotic approximations?Begin with idea of a sharp null: Y1i = Y0i ∀i . (Gerber and Green, p.62)

• If Y1i = Y0i ∀i , then if we observe either, we have seen both.

• All possible treatment arrangements would yield the same Y values.

• We could then calculate all possible treatment effect estimates underthe sharp null.

• The distribution of these possible treatment effects allows us tocompute p-values: The probability that, under the null, somethingthis large or larger would occur at random. (For the two sided test,“large” means in absolute value terms.)

• This extends naturally to the case where treatment assignments arerestricted in some way. Recall, for example, the Bruhn andMcKenzie (2009) discussion of the many different restrictions thatcan be used to yield balanced randomization.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 4

Page 8: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

How can we do hypothesis testing without asymptotic approximations?Begin with idea of a sharp null: Y1i = Y0i ∀i . (Gerber and Green, p.62)

• If Y1i = Y0i ∀i , then if we observe either, we have seen both.

• All possible treatment arrangements would yield the same Y values.

• We could then calculate all possible treatment effect estimates underthe sharp null.

• The distribution of these possible treatment effects allows us tocompute p-values: The probability that, under the null, somethingthis large or larger would occur at random. (For the two sided test,“large” means in absolute value terms.)

• This extends naturally to the case where treatment assignments arerestricted in some way. Recall, for example, the Bruhn andMcKenzie (2009) discussion of the many different restrictions thatcan be used to yield balanced randomization.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 4

Page 9: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

How can we do hypothesis testing without asymptotic approximations?Begin with idea of a sharp null: Y1i = Y0i ∀i . (Gerber and Green, p.62)

• If Y1i = Y0i ∀i , then if we observe either, we have seen both.

• All possible treatment arrangements would yield the same Y values.

• We could then calculate all possible treatment effect estimates underthe sharp null.

• The distribution of these possible treatment effects allows us tocompute p-values:

The probability that, under the null, somethingthis large or larger would occur at random. (For the two sided test,“large” means in absolute value terms.)

• This extends naturally to the case where treatment assignments arerestricted in some way. Recall, for example, the Bruhn andMcKenzie (2009) discussion of the many different restrictions thatcan be used to yield balanced randomization.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 4

Page 10: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

How can we do hypothesis testing without asymptotic approximations?Begin with idea of a sharp null: Y1i = Y0i ∀i . (Gerber and Green, p.62)

• If Y1i = Y0i ∀i , then if we observe either, we have seen both.

• All possible treatment arrangements would yield the same Y values.

• We could then calculate all possible treatment effect estimates underthe sharp null.

• The distribution of these possible treatment effects allows us tocompute p-values: The probability that, under the null, somethingthis large or larger would occur at random. (For the two sided test,“large” means in absolute value terms.)

• This extends naturally to the case where treatment assignments arerestricted in some way. Recall, for example, the Bruhn andMcKenzie (2009) discussion of the many different restrictions thatcan be used to yield balanced randomization.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 4

Page 11: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

How can we do hypothesis testing without asymptotic approximations?Begin with idea of a sharp null: Y1i = Y0i ∀i . (Gerber and Green, p.62)

• If Y1i = Y0i ∀i , then if we observe either, we have seen both.

• All possible treatment arrangements would yield the same Y values.

• We could then calculate all possible treatment effect estimates underthe sharp null.

• The distribution of these possible treatment effects allows us tocompute p-values: The probability that, under the null, somethingthis large or larger would occur at random. (For the two sided test,“large” means in absolute value terms.)

• This extends naturally to the case where treatment assignments arerestricted in some way. Recall, for example, the Bruhn andMcKenzie (2009) discussion of the many different restrictions thatcan be used to yield balanced randomization.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 4

Page 12: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

It is often impractical to enumerate all possible treatment effects.Instead, we sample a large number of them:

• Regress Y on T. Note the absolute value of the coefficient on T.

• For a large number of iterations:I Devise an alternative random assignment of treatment. In the

simplest, unrestricted case, this means scrambling the relationshipbetween Y and T randomly, preserving the number of treatment andcomparison units in T. Call this assignment AlternativeTreatment.

I Regress Y on AlternativeTreatment.

I Note whether the absolute value of the coefficient onAlternativeTreatment equals or exceeds the absolute value of theoriginal (true) coefficient on T. If so, increment a counter.

• Divide the counter by the number of iterations. You have a p-value!

Gerber and Green, p.63: “... the calculation of p-values based on aninventory of possible randomizations is called randomization inference.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 5

Page 13: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

It is often impractical to enumerate all possible treatment effects.Instead, we sample a large number of them:

• Regress Y on T. Note the absolute value of the coefficient on T.

• For a large number of iterations:I Devise an alternative random assignment of treatment. In the

simplest, unrestricted case, this means scrambling the relationshipbetween Y and T randomly, preserving the number of treatment andcomparison units in T. Call this assignment AlternativeTreatment.

I Regress Y on AlternativeTreatment.

I Note whether the absolute value of the coefficient onAlternativeTreatment equals or exceeds the absolute value of theoriginal (true) coefficient on T. If so, increment a counter.

• Divide the counter by the number of iterations. You have a p-value!

Gerber and Green, p.63: “... the calculation of p-values based on aninventory of possible randomizations is called randomization inference.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 5

Page 14: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

It is often impractical to enumerate all possible treatment effects.Instead, we sample a large number of them:

• Regress Y on T. Note the absolute value of the coefficient on T.

• For a large number of iterations:I Devise an alternative random assignment of treatment. In the

simplest, unrestricted case, this means scrambling the relationshipbetween Y and T randomly, preserving the number of treatment andcomparison units in T. Call this assignment AlternativeTreatment.

I Regress Y on AlternativeTreatment.

I Note whether the absolute value of the coefficient onAlternativeTreatment equals or exceeds the absolute value of theoriginal (true) coefficient on T. If so, increment a counter.

• Divide the counter by the number of iterations. You have a p-value!

Gerber and Green, p.63: “... the calculation of p-values based on aninventory of possible randomizations is called randomization inference.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 5

Page 15: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

It is often impractical to enumerate all possible treatment effects.Instead, we sample a large number of them:

• Regress Y on T. Note the absolute value of the coefficient on T.

• For a large number of iterations:I Devise an alternative random assignment of treatment. In the

simplest, unrestricted case, this means scrambling the relationshipbetween Y and T randomly, preserving the number of treatment andcomparison units in T. Call this assignment AlternativeTreatment.

I Regress Y on AlternativeTreatment.

I Note whether the absolute value of the coefficient onAlternativeTreatment equals or exceeds the absolute value of theoriginal (true) coefficient on T. If so, increment a counter.

• Divide the counter by the number of iterations. You have a p-value!

Gerber and Green, p.63: “... the calculation of p-values based on aninventory of possible randomizations is called randomization inference.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 5

Page 16: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

It is often impractical to enumerate all possible treatment effects.Instead, we sample a large number of them:

• Regress Y on T. Note the absolute value of the coefficient on T.

• For a large number of iterations:I Devise an alternative random assignment of treatment. In the

simplest, unrestricted case, this means scrambling the relationshipbetween Y and T randomly, preserving the number of treatment andcomparison units in T. Call this assignment AlternativeTreatment.

I Regress Y on AlternativeTreatment.

I Note whether the absolute value of the coefficient onAlternativeTreatment equals or exceeds the absolute value of theoriginal (true) coefficient on T. If so, increment a counter.

• Divide the counter by the number of iterations. You have a p-value!

Gerber and Green, p.63: “... the calculation of p-values based on aninventory of possible randomizations is called randomization inference.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 5

Page 17: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

It is often impractical to enumerate all possible treatment effects.Instead, we sample a large number of them:

• Regress Y on T. Note the absolute value of the coefficient on T.

• For a large number of iterations:I Devise an alternative random assignment of treatment. In the

simplest, unrestricted case, this means scrambling the relationshipbetween Y and T randomly, preserving the number of treatment andcomparison units in T. Call this assignment AlternativeTreatment.

I Regress Y on AlternativeTreatment.

I Note whether the absolute value of the coefficient onAlternativeTreatment equals or exceeds the absolute value of theoriginal (true) coefficient on T. If so, increment a counter.

• Divide the counter by the number of iterations. You have a p-value!

Gerber and Green, p.63: “... the calculation of p-values based on aninventory of possible randomizations is called randomization inference.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 5

Page 18: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

Gerber and Green, p.64:

• “The sampling distribution of the test statistic under the nullhypothesis is computed by simulating all possible randomassignments. When the number of random assignments is too largeto simulate, the sampling distribution may be approximated by alarge random sample of possible assignments. p-values arecalculated by comparing the observed test statistic to thedistribution of test statistics under the null hypothesis.”

NOTE: How large a random sample?What is the standard deviation of a binary outcome with mean 0.05?About 0.22.Standard error (of this estimated p-value) ...in a sample of size 100alternative randomizations? About 0.022....in a sample of size 10,000 alternative randomizations? About 0.0022.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 6

Page 19: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

Gerber and Green, p.64:

• “The sampling distribution of the test statistic under the nullhypothesis is computed by simulating all possible randomassignments. When the number of random assignments is too largeto simulate, the sampling distribution may be approximated by alarge random sample of possible assignments. p-values arecalculated by comparing the observed test statistic to thedistribution of test statistics under the null hypothesis.”

NOTE: How large a random sample?What is the standard deviation of a binary outcome with mean 0.05?

About 0.22.Standard error (of this estimated p-value) ...in a sample of size 100alternative randomizations? About 0.022....in a sample of size 10,000 alternative randomizations? About 0.0022.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 6

Page 20: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

Gerber and Green, p.64:

• “The sampling distribution of the test statistic under the nullhypothesis is computed by simulating all possible randomassignments. When the number of random assignments is too largeto simulate, the sampling distribution may be approximated by alarge random sample of possible assignments. p-values arecalculated by comparing the observed test statistic to thedistribution of test statistics under the null hypothesis.”

NOTE: How large a random sample?What is the standard deviation of a binary outcome with mean 0.05?About 0.22.

Standard error (of this estimated p-value) ...in a sample of size 100alternative randomizations? About 0.022....in a sample of size 10,000 alternative randomizations? About 0.0022.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 6

Page 21: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

Gerber and Green, p.64:

• “The sampling distribution of the test statistic under the nullhypothesis is computed by simulating all possible randomassignments. When the number of random assignments is too largeto simulate, the sampling distribution may be approximated by alarge random sample of possible assignments. p-values arecalculated by comparing the observed test statistic to thedistribution of test statistics under the null hypothesis.”

NOTE: How large a random sample?What is the standard deviation of a binary outcome with mean 0.05?About 0.22.Standard error (of this estimated p-value) ...in a sample of size 100alternative randomizations?

About 0.022....in a sample of size 10,000 alternative randomizations? About 0.0022.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 6

Page 22: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

Gerber and Green, p.64:

• “The sampling distribution of the test statistic under the nullhypothesis is computed by simulating all possible randomassignments. When the number of random assignments is too largeto simulate, the sampling distribution may be approximated by alarge random sample of possible assignments. p-values arecalculated by comparing the observed test statistic to thedistribution of test statistics under the null hypothesis.”

NOTE: How large a random sample?What is the standard deviation of a binary outcome with mean 0.05?About 0.22.Standard error (of this estimated p-value) ...in a sample of size 100alternative randomizations? About 0.022.

...in a sample of size 10,000 alternative randomizations? About 0.0022.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 6

Page 23: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

Gerber and Green, p.64:

• “The sampling distribution of the test statistic under the nullhypothesis is computed by simulating all possible randomassignments. When the number of random assignments is too largeto simulate, the sampling distribution may be approximated by alarge random sample of possible assignments. p-values arecalculated by comparing the observed test statistic to thedistribution of test statistics under the null hypothesis.”

NOTE: How large a random sample?What is the standard deviation of a binary outcome with mean 0.05?About 0.22.Standard error (of this estimated p-value) ...in a sample of size 100alternative randomizations? About 0.022....in a sample of size 10,000 alternative randomizations?

About 0.0022.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 6

Page 24: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference

Gerber and Green, p.64:

• “The sampling distribution of the test statistic under the nullhypothesis is computed by simulating all possible randomassignments. When the number of random assignments is too largeto simulate, the sampling distribution may be approximated by alarge random sample of possible assignments. p-values arecalculated by comparing the observed test statistic to thedistribution of test statistics under the null hypothesis.”

NOTE: How large a random sample?What is the standard deviation of a binary outcome with mean 0.05?About 0.22.Standard error (of this estimated p-value) ...in a sample of size 100alternative randomizations? About 0.022....in a sample of size 10,000 alternative randomizations? About 0.0022.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 6

Page 25: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference Confidence Intervals

Major drawback

• This doesn’t give you a confidence interval automatically.

• Under assumptions, you can construct them (Gerber and Green,section 3.5):

I “The most traightforward method for filling in missing potentialoutcomes is to assume that the treatment effect τi is the same for allsubjects.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 7

Page 26: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference Confidence Intervals

Gerber and Green, section 3.5:

• “For subjects in the control condition, missing Yi (1) values areimputed by adding the estimated ATE to the observed values ofYi (0).”

• “Similarly, for subjects in the treatment condition, missing Yi (0)values are imputed by subtracting the estimated ATE from theobserved values of Yi (1).”

• “This approach yields a complete schedule of potential outcomes,which we may then use to simulate all possible random allocations.”

• “In order to form a 95% confidence interval, we list the estimatedATE from each random allocation in ascending order. The estimateat the 2.5th percentile marks the bottom of the interval, and theestimate at the 97.5th percentile marks the top of the interval.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 8

Page 27: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference Confidence Intervals

Gerber and Green, section 3.5:

• “For subjects in the control condition, missing Yi (1) values areimputed by adding the estimated ATE to the observed values ofYi (0).”

• “Similarly, for subjects in the treatment condition, missing Yi (0)values are imputed by subtracting the estimated ATE from theobserved values of Yi (1).”

• “This approach yields a complete schedule of potential outcomes,which we may then use to simulate all possible random allocations.”

• “In order to form a 95% confidence interval, we list the estimatedATE from each random allocation in ascending order. The estimateat the 2.5th percentile marks the bottom of the interval, and theestimate at the 97.5th percentile marks the top of the interval.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 8

Page 28: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference Confidence Intervals

Gerber and Green, section 3.5:

• “For subjects in the control condition, missing Yi (1) values areimputed by adding the estimated ATE to the observed values ofYi (0).”

• “Similarly, for subjects in the treatment condition, missing Yi (0)values are imputed by subtracting the estimated ATE from theobserved values of Yi (1).”

• “This approach yields a complete schedule of potential outcomes,which we may then use to simulate all possible random allocations.”

• “In order to form a 95% confidence interval, we list the estimatedATE from each random allocation in ascending order. The estimateat the 2.5th percentile marks the bottom of the interval, and theestimate at the 97.5th percentile marks the top of the interval.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 8

Page 29: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference Confidence Intervals

Gerber and Green, section 3.5:

• “For subjects in the control condition, missing Yi (1) values areimputed by adding the estimated ATE to the observed values ofYi (0).”

• “Similarly, for subjects in the treatment condition, missing Yi (0)values are imputed by subtracting the estimated ATE from theobserved values of Yi (1).”

• “This approach yields a complete schedule of potential outcomes,which we may then use to simulate all possible random allocations.”

• “In order to form a 95% confidence interval, we list the estimatedATE from each random allocation in ascending order. The estimateat the 2.5th percentile marks the bottom of the interval, and theestimate at the 97.5th percentile marks the top of the interval.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 8

Page 30: ECON 626: Applied Microeconomics - Economics.ozier.com

Randomization Inference Confidence Intervals

Gerber and Green, section 3.5:

• “For subjects in the control condition, missing Yi (1) values areimputed by adding the estimated ATE to the observed values ofYi (0).”

• “Similarly, for subjects in the treatment condition, missing Yi (0)values are imputed by subtracting the estimated ATE from theobserved values of Yi (1).”

• “This approach yields a complete schedule of potential outcomes,which we may then use to simulate all possible random allocations.”

• “In order to form a 95% confidence interval, we list the estimatedATE from each random allocation in ascending order. The estimateat the 2.5th percentile marks the bottom of the interval, and theestimate at the 97.5th percentile marks the top of the interval.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 8

Page 31: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(

84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 32: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(

84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 33: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(

84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 34: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(

84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 35: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(

84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 36: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)

= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 37: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 =

= 16 + 16 + 1 + 1 = 34 out of...(84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 38: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 =

34 out of...(84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 39: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34

out of...(84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 40: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(

84

)=

70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 41: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(

84

)= 70.

p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 42: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(

84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 43: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 1

Remember the Lady Tasting Tea from the first class?Suppose she gets a certain number right. For example:

• Eight cups, four of which had milk added first.

• After tasting, suppose she correctly says there are four cups whichhad milk added first, but while she correctly identifies three cups,she gets one wrong. What is the probability of being that correlatedwith the truth, or better? (in absolute value terms?)

•(

43

)(41

)+

(41

)(43

)+

(44

)(40

)+

(40

)(44

)= 4 · 4 + 4 · 4 + 1 · 1 + 1 · 1 = = 16 + 16 + 1 + 1 = 34 out of...(

84

)= 70. p-value just under 50 percent.

Or what if it were ten cups, and she got 4 out of 5?This becomes unwieldy to calculate exactly. Activity: randomly sample.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 9

Page 44: ECON 626: Applied Microeconomics - Economics.ozier.com

Part IIa: Bootstrap

Page 45: ECON 626: Applied Microeconomics - Economics.ozier.com

Bootstrap basics

• See Angrist and Pischke, pp.300-301 (Bootstrap).

• Sampling {Yi ,Xi} with replacement: “pairs bootstrap” or“nonparametric bootstrap.”

• Keeping Xi fixed, sampling ei with replacement, constructing newoutcomes Yi treating Xi as fixed using the original β: one kind of“parametric bootstrap.”

• Keeping Xi fixed, constructing new outcomes Yi treating Xi as fixedusing the original β, but randomly flipping the sign of ei , preservingrelationships between Xi and the variance of the residual: “wildbootstrap.”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 11

Page 46: ECON 626: Applied Microeconomics - Economics.ozier.com

Part IIb:Few Clusters;

Wild Cluster Bootstrap

Page 47: ECON 626: Applied Microeconomics - Economics.ozier.com

What is the problem with having too few clusters?

0.2

.4.6

.81

CD

F o

f p-v

alue

s

0 .2 .4 .6 .8 1p-values

Ideal

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 13

Page 48: ECON 626: Applied Microeconomics - Economics.ozier.com

What is the problem with having too few clusters?

0.2

.4.6

.81

CD

F o

f p-v

alue

s

0 .2 .4 .6 .8 1p-values

Ideal Six uneven clusters

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 13

Page 49: ECON 626: Applied Microeconomics - Economics.ozier.com

How will “bootstrapping” solve the cluster problem?

Bootstrapping is drawing (often with replacement) from some aspect ofthe data to quantify variability of a statistic.

We cluster standard errors because we are concerned that the error maybe heteroskedastic and correlated within clusters. So we could notsensibly use a bootstrapping procedure that ignored covariates or thecluster structure. Cameron, Gelbach, Miller (2008) and Cameron andMiller (2015) discuss a procedure that respects covariate structure(“wild”) and cluster structure (“cluster”) while drawing alternativeresiduals (“bootstrap”).

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 14

Page 50: ECON 626: Applied Microeconomics - Economics.ozier.com

How will “bootstrapping” solve the cluster problem?

Bootstrapping is drawing (often with replacement) from some aspect ofthe data to quantify variability of a statistic.We cluster standard errors because we are concerned that the error maybe heteroskedastic and correlated within clusters. So we could notsensibly use a bootstrapping procedure that ignored covariates or thecluster structure.

Cameron, Gelbach, Miller (2008) and Cameron andMiller (2015) discuss a procedure that respects covariate structure(“wild”) and cluster structure (“cluster”) while drawing alternativeresiduals (“bootstrap”).

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 14

Page 51: ECON 626: Applied Microeconomics - Economics.ozier.com

How will “bootstrapping” solve the cluster problem?

Bootstrapping is drawing (often with replacement) from some aspect ofthe data to quantify variability of a statistic.We cluster standard errors because we are concerned that the error maybe heteroskedastic and correlated within clusters. So we could notsensibly use a bootstrapping procedure that ignored covariates or thecluster structure. Cameron, Gelbach, Miller (2008) and Cameron andMiller (2015) discuss a procedure that respects covariate structure(“wild”)

and cluster structure (“cluster”) while drawing alternativeresiduals (“bootstrap”).

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 14

Page 52: ECON 626: Applied Microeconomics - Economics.ozier.com

How will “bootstrapping” solve the cluster problem?

Bootstrapping is drawing (often with replacement) from some aspect ofthe data to quantify variability of a statistic.We cluster standard errors because we are concerned that the error maybe heteroskedastic and correlated within clusters. So we could notsensibly use a bootstrapping procedure that ignored covariates or thecluster structure. Cameron, Gelbach, Miller (2008) and Cameron andMiller (2015) discuss a procedure that respects covariate structure(“wild”) and cluster structure (“cluster”)

while drawing alternativeresiduals (“bootstrap”).

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 14

Page 53: ECON 626: Applied Microeconomics - Economics.ozier.com

How will “bootstrapping” solve the cluster problem?

Bootstrapping is drawing (often with replacement) from some aspect ofthe data to quantify variability of a statistic.We cluster standard errors because we are concerned that the error maybe heteroskedastic and correlated within clusters. So we could notsensibly use a bootstrapping procedure that ignored covariates or thecluster structure. Cameron, Gelbach, Miller (2008) and Cameron andMiller (2015) discuss a procedure that respects covariate structure(“wild”) and cluster structure (“cluster”) while drawing alternativeresiduals (“bootstrap”).

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 14

Page 54: ECON 626: Applied Microeconomics - Economics.ozier.com

The Wild Cluster Bootstrap

Cameron, Gelbach, and Miller procedure goes as follows (Cameron andMiller 2015, Section VI.C.2):

• “First, estimate the main model, imposing (forcing) the nullhypothesis that you wish to test... For example, for test of statisticalsignificance of a single variable regress yig on all components of xigexcept the variable that has regressor with coefficient zero under thenull hypothesis.”

• “Form the residual uig = yig − x ′ig βH0”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 15

Page 55: ECON 626: Applied Microeconomics - Economics.ozier.com

The Wild Cluster Bootstrap

Cameron, Gelbach, and Miller procedure goes as follows (Cameron andMiller 2015, Section VI.C.2):

• “First, estimate the main model, imposing (forcing) the nullhypothesis that you wish to test... For example, for test of statisticalsignificance of a single variable regress yig on all components of xigexcept the variable that has regressor with coefficient zero under thenull hypothesis.”

• “Form the residual uig = yig − x ′ig βH0”

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 15

Page 56: ECON 626: Applied Microeconomics - Economics.ozier.com

The Wild Cluster Bootstrap

Cameron, Gelbach, and Miller procedure goes as follows (Cameron andMiller 2015, Section VI.C.2):

• In each resampling:

I “Randomly assign cluster g the weight dg = −1 with probability 0.5and the weight dg = 1 with probability 0.5. All observations incluster g get the same value of the weight.” (Rademacher weights)

I “Generate new pseudo-residuals u∗ig = dg × uig , and hence new

outcome variables y∗ig = x ′ig βH0 + u∗ig . Then proceed with step 2 asbefore, regressing y∗ig on xig [not imposing the null], and calculate w∗

[the t-statistic from this regression, with clustered standard errors.]”

• The p-value for the the test based on the original sample statistic wequals the proportion of times that |w | > |w∗b |.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 16

Page 57: ECON 626: Applied Microeconomics - Economics.ozier.com

The Wild Cluster Bootstrap

Cameron, Gelbach, and Miller procedure goes as follows (Cameron andMiller 2015, Section VI.C.2):

• In each resampling:I “Randomly assign cluster g the weight dg = −1 with probability 0.5

and the weight dg = 1 with probability 0.5. All observations incluster g get the same value of the weight.” (Rademacher weights)

I “Generate new pseudo-residuals u∗ig = dg × uig , and hence new

outcome variables y∗ig = x ′ig βH0 + u∗ig . Then proceed with step 2 asbefore, regressing y∗ig on xig [not imposing the null], and calculate w∗

[the t-statistic from this regression, with clustered standard errors.]”

• The p-value for the the test based on the original sample statistic wequals the proportion of times that |w | > |w∗b |.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 16

Page 58: ECON 626: Applied Microeconomics - Economics.ozier.com

The Wild Cluster Bootstrap

Cameron, Gelbach, and Miller procedure goes as follows (Cameron andMiller 2015, Section VI.C.2):

• In each resampling:I “Randomly assign cluster g the weight dg = −1 with probability 0.5

and the weight dg = 1 with probability 0.5. All observations incluster g get the same value of the weight.” (Rademacher weights)

I “Generate new pseudo-residuals u∗ig = dg × uig , and hence new

outcome variables y∗ig = x ′ig βH0 + u∗ig . Then proceed with step 2 asbefore, regressing y∗ig on xig [not imposing the null], and calculate w∗

[the t-statistic from this regression, with clustered standard errors.]”

• The p-value for the the test based on the original sample statistic wequals the proportion of times that |w | > |w∗b |.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 16

Page 59: ECON 626: Applied Microeconomics - Economics.ozier.com

The Wild Cluster Bootstrap

Cameron, Gelbach, and Miller procedure goes as follows (Cameron andMiller 2015, Section VI.C.2):

• In each resampling:I “Randomly assign cluster g the weight dg = −1 with probability 0.5

and the weight dg = 1 with probability 0.5. All observations incluster g get the same value of the weight.” (Rademacher weights)

I “Generate new pseudo-residuals u∗ig = dg × uig , and hence new

outcome variables y∗ig = x ′ig βH0 + u∗ig . Then proceed with step 2 asbefore, regressing y∗ig on xig [not imposing the null], and calculate w∗

[the t-statistic from this regression, with clustered standard errors.]”

• The p-value for the the test based on the original sample statistic wequals the proportion of times that |w | > |w∗b |.

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 16

Page 60: ECON 626: Applied Microeconomics - Economics.ozier.com

What happens with six clusters

0.2

.4.6

.81

CD

F o

f p-v

alue

s

0 .2 .4 .6 .8 1p-values

Ideal Six uneven clusters

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 17

Page 61: ECON 626: Applied Microeconomics - Economics.ozier.com

What happens with six clusters

0.2

.4.6

.81

CD

F o

f p-v

alue

s

0 .2 .4 .6 .8 1p-values

Ideal Six uneven clusters WCB Rademacher

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 17

Page 62: ECON 626: Applied Microeconomics - Economics.ozier.com

So-called Rademacher and Webb weights

017

3350

6783

100

Per

cent

-2 -1 0 1 2dr

Rademacher weights

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 18

Page 63: ECON 626: Applied Microeconomics - Economics.ozier.com

So-called Rademacher and Webb weights

017

3350

6783

100

Per

cent

-2 -1 0 1 2dr

Rademacher weights

017

3350

6783

100

Per

cent

-2 -1 0 1 2dw

Webb weights

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 18

Page 64: ECON 626: Applied Microeconomics - Economics.ozier.com

What happens with six clusters

0.2

.4.6

.81

CD

F o

f p-v

alue

s

0 .2 .4 .6 .8 1p-values

Ideal Six uneven clusters WCB Rademacher

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 19

Page 65: ECON 626: Applied Microeconomics - Economics.ozier.com

What happens with six clusters

0.2

.4.6

.81

CD

F o

f p-v

alue

s

0 .2 .4 .6 .8 1p-values

Ideal Six uneven clusters WCB Rademacher WCB Webb

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 19

Page 66: ECON 626: Applied Microeconomics - Economics.ozier.com

What happens with eight clusters

0.2

.4.6

.81

CD

F o

f p-v

alue

s

0 .2 .4 .6 .8 1p-values

Ideal Eight uneven clusters WCB Rademacher WCB Webb

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 20

Page 67: ECON 626: Applied Microeconomics - Economics.ozier.com

What happens with six clusters (zoom)

0.1

.2.3

.4.5

CD

F o

f p-v

alue

s

0 .05 .1 .15 .2p-values

Ideal Six uneven clusters WCB Rademacher WCB Webb

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 21

Page 68: ECON 626: Applied Microeconomics - Economics.ozier.com

What happens with eight clusters (zoom)

0.1

.2.3

.4.5

CD

F o

f p-v

alue

s

0 .05 .1 .15 .2p-values

Ideal Eight uneven clusters WCB Rademacher WCB Webb

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 22

Page 69: ECON 626: Applied Microeconomics - Economics.ozier.com

Activity 2

UMD Economics 626: Applied Microeconomics Lecture 8: Permutation and Bootstrap, Slide 23