The Power of Pairing in Cluster Randomized Block Designs: A Monte Carlo Simulation
description
Transcript of The Power of Pairing in Cluster Randomized Block Designs: A Monte Carlo Simulation
The Power of Pairing in Cluster Randomized Block Designs: A Monte Carlo Simulation
Nianbo Dong & Mark Lipsey
03/04/2010
Power Analysis for Group-Randomized Experiments
1. Two Big Design Families (Kirk, 1995)
1) Hierarchical Design
2) Cluster Randomized Block Design
2. Three Ways to do Power Analysis
1) Software, e.g., Optimal Design 2.0 (Spybrook, Raudenbush, Congdon, & Martinez, 2009)
2) MDES formula (Bloom, 2006; Schochet, 2008)
3) Power table using operational effect size(Hedges, 2009; Konstantopoulos, 2009)
2
Matched-Pair Cluster-Randomized Design (1)
1. Advantages1) Avoiding bad randomization, 2) Face validity
2. Cost: Loss of degree of freedom
3
3. But, the gain in predictive power may outweigh the loss of degrees of freedom (Billewicz, 1965; Bloom, 2007; Hedges, 2009; Martin, Diehr, Perrin, & Koepsell, 1993; Raudenbush, Martinez, & Spybrook, 2007)
4. Break-even point using MDES for fixed pair effect model (Bloom, 2007)
Matched-Pair Cluster-Randomized Design (2)
5. MDES Comparison
4
MDES for 2-level hierarchical design, w/o covariance adjustment
(Bloom, 2006)
MDES for 3-level matched-pair cluster-randomized design, random pair effect model w/o covariance adjustment, VC
JnJ
RMMDES B
J)-1(4)1(4 *2*
12
M (x) = / 2t + 1t
J: # of clusters; n: average # of individuals
* : ICC for hierarchical design;
: Pair-level ICC for matched-pair cluster-randomized design
3
(SE from Raudenbush & Liu, 2000)
JnJMMDES J
)-1(44 **
2
*32
BR
Matched-Pair Cluster-Randomized Design (2)
5. MDES Comparison
5
MDES for 2-level hierarchical design, w/o covariance adjustment
(Bloom, 2006)
MDES for 3-level matched-pair cluster-randomized design, random pair effect model w/o covariance adjustment, VC
JnJ
RMMDES B
J)-1(4)1(4 *2*
12
M (x) = / 2t + 1t
J: # of clusters; n: average # of individuals
* : ICC for hierarchical design;
: Pair-level ICC for matched-pair cluster-randomized design
3
(SE from Raudenbush & Liu, 2000)
JnJMMDES J
)-1(44 **
2
*32
BR
6
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adjFixed pair effect
With cov-adj
Ignoring pair With cov-adj
YesW/o cov-adj
Random pair effect
With cov-adj
W/o cov-adjFixed pair effect
With cov-adj
Ignoring pair With cov-adj
No With cov-adj
Able to make close matching
Pre-randomization
Matching Unable to make close matching
L1 L2 L3 L4
Conceptual Model: Options & Decisions
7
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adjFixed pair effect
With cov-adj
Ignoring pair With cov-adj
YesW/o cov-adj
Random pair effect
With cov-adj
W/o cov-adjFixed pair effect
With cov-adj
Ignoring pair With cov-adj
No With cov-adj
Able to make close matching
Pre-randomization
Matching Unable to make close matching
L1 L2 L3 L4
Conceptual Model: Options & Decisions
Research Questions
The Overall QuestionAre there design and analysis options other than increasing the sample size that might keep pre-randomization matching from degrading power relative to the analogous unmatched design?
8
Four Sub-Questions
9
How much difference does it make to statistical power:
1. If we are able to make close matches or unable to do so?2. If we are treating the pairwise blocks as fixed effects vs. random effects?3. If we ignore the pairwise blocking entirely (and does this compromise the Type I error rate)?4. If we also use the blocking variable as a covariate in the analysis?
Simulation: Parameter Combinations
10
R2 = 0.2
R2 = 0.5
R2 = 0.2
R2 = 0.5
R2 = 0.2
R2 = 0.5
10
30
60
10
30
60
Note . n =20
ESV=0.01
0.2
J
0.1
ICC
ES=0, 0.3, 0.5
ESV=0 ESV=0.05
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.75/.13Ignoring pair With cov-adj
Yes
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.76/.13Ignoring pair With cov-adj
No With cov-adj
Pre-randomization
Matching Unable to make close matching
.74/.14
.66/.08
.71/.08
.66/.10
.70/.08Able to make close matching
.71/.12
.63/.08
.69/.12
.75/.13
.69/.12
.71/.08
.75/.13
.58/.10
.77/.13
.55/.09
.73/.08
.76/.13
.74/.14
.69/.08
Results: Median and Minimum Power on Each Branch
11
Median/MinSE =.009-.016
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.75/.13Ignoring pair With cov-adj
Yes
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.76/.13Ignoring pair With cov-adj
No With cov-adj
Pre-randomization
Matching Unable to make close matching
.74/.14
.66/.08
.71/.08
.66/.10
.70/.08Able to make close matching
.71/.12
.63/.08
.69/.12
.75/.13
.69/.12
.71/.08
.75/.13
.58/.10
.77/.13
.55/.09
.73/.08
.76/.13
.74/.14
.69/.08
Results: Median and Minimum Power on Each Branch
12
Median/MinSE =.009-.016
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.75/.13Ignoring pair With cov-adj
Yes
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.76/.13Ignoring pair With cov-adj
No With cov-adj
Pre-randomization
Matching Unable to make close matching
.74/.14
.66/.08
.71/.08
.66/.10
.70/.08Able to make close matching
.71/.12
.63/.08
.69/.12
.75/.13
.69/.12
.71/.08
.75/.13
.58/.10
.77/.13
.55/.09
.73/.08
.76/.13
.74/.14
.69/.08
Results: Median and Minimum Power on Each Branch
13
Median/MinSE =.009-.016
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.75/.13Ignoring pair With cov-adj
Yes
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.76/.13Ignoring pair With cov-adj
No With cov-adj
Pre-randomization
Matching Unable to make close matching
.74/.14
.66/.08
.71/.08
.66/.10
.70/.08Able to make close matching
.71/.12
.63/.08
.69/.12
.75/.13
.69/.12
.71/.08
.75/.13
.58/.10
.77/.13
.55/.09
.73/.08
.76/.13
.74/.14
.69/.08
Results: Median and Minimum Power on Each Branch
14
Median/MinSE =.009-.016
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.75/.13Ignoring pair With cov-adj
Yes
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.76/.13Ignoring pair With cov-adj
No With cov-adj
Pre-randomization
Matching Unable to make close matching
.74/.14
.66/.08
.71/.08
.66/.10
.70/.08Able to make close matching
.71/.12
.63/.08
.69/.12
.75/.13
.69/.12
.71/.08
.75/.13
.58/.10
.77/.13
.55/.09
.73/.08
.76/.13
.74/.14
.69/.08
Results: Median and Minimum Power on Each Branch
15
Median/MinSE =.009-.016
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.75/.13Ignoring pair With cov-adj
Yes
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.76/.13Ignoring pair With cov-adj
No With cov-adj
Pre-randomization
Matching Unable to make close matching
.74/.14
.66/.08
.71/.08
.66/.10
.70/.08Able to make close matching
.71/.12
.63/.08
.69/.12
.75/.13
.69/.12
.71/.08
.75/.13
.58/.10
.77/.13
.55/.09
.73/.08
.76/.13
.74/.14
.69/.08
Results: Median and Minimum Power on Each Branch
16
Median/MinSE =.009-.016
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.75/.13Ignoring pair With cov-adj
Yes
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.76/.13Ignoring pair With cov-adj
No With cov-adj
Pre-randomization
Matching Unable to make close matching
.74/.14
.66/.08
.71/.08
.66/.10
.70/.08Able to make close matching
.71/.12
.63/.08
.69/.12
.75/.13
.69/.12
.71/.08
.75/.13
.58/.10
.77/.13
.55/.09
.73/.08
.76/.13
.74/.14
.69/.08
Results: Median and Minimum Power on Each Branch
17
Median/MinSE =.009-.016
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.75/.13Ignoring pair With cov-adj
Yes
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.76/.13Ignoring pair With cov-adj
No With cov-adj
Pre-randomization
Matching Unable to make close matching
.74/.14
.66/.08
.71/.08
.66/.10
.70/.08Able to make close matching
.71/.12
.63/.08
.69/.12
.75/.13
.69/.12
.71/.08
.75/.13
.58/.10
.77/.13
.55/.09
.73/.08
.76/.13
.74/.14
.69/.08
Results: Median and Minimum Power on Each Branch
18
Median/MinSE =.009-.016
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.75/.13Ignoring pair With cov-adj
Yes
W/o cov-adjRandom pair
effect
With cov-adj
W/o cov-adj
Fixed pair effect
With cov-adj
.76/.13Ignoring pair With cov-adj
No With cov-adj
Pre-randomization
Matching Unable to make close matching
.74/.14
.66/.08
.71/.08
.66/.10
.70/.08Able to make close matching
.71/.12
.63/.08
.69/.12
.75/.13
.69/.12
.71/.08
.75/.13
.58/.10
.77/.13
.55/.09
.73/.08
.76/.13
.74/.14
.69/.08
Results: Median and Minimum Power on Each Branch
19
Median/MinSE =.009-.016
Conclusions•The advantages of pre-randomization matching do not have to come at the cost of reduced power– even when the matching is not very good.
20
•The most important technique for maintaining power is to also use the matching variable as a covariate.
•The random effects model does not necessarily have less power than the fixed effects alternative.
•Ignoring the pairwise blocking variable in the analysis, though not faithful to the actual design used, does not appear to cause problems with either the Type I or Type II error rate. (Consistent with Diehr, Martin, Koepsell, & Cheadle, 1995; Lynn & McCulloch, 1992; Proschan, 1996)
Thanks
21
22
Statistical Power from Simulation (J=30, n=20, ES=.3, 2
1R = 22R =.5, = .05 for two-tailed test)
Combination ESV = 0 ESV = .05
ICC = .1 ICC = .2 ICC = .1 ICC = .2 1 .81 .61 .70 .54 2 .69 .53 .58 .46 3 .78 .57 .64 .47 4 .70 .53 .58 .46 5 .85 .63 .71 .55 6 .51 .36 .45 .34 7 .78 .58 .64 .49 8 .56 .39 .49 .37 9 .82 .63 .70 .55
10 .84 .62 .69 .53 11 .81 .63 .69 .54
Appendix 1:
23
Appendix 2:Statistical Power from Simulation (J=10, n=20, ES=.3, 2
1R = 22R =.2, = .05 for two-tailed test)
Combination ESV = 0 ESV = .05
ICC = .1 ICC = .2 ICC = .1 ICC = .2 1 .19 .14 .19 .14 2 .15 .13 .15 .12 3 .09 .09 .11 .10 4 .15 .12 .15 .13 5 .18 .14 .16 .14 6 .14 .12 .13 .09 7 .10 .11 .10 .08 8 .16 .13 .14 .10 9 .17 .15 .15 .13
10 .22 .15 .20 .15 11 .22 .15 .18 .13
Appendix 3: MDES for Matched-Pair Cluster-Randomized Design
24
Random Pair Effect Model - Unconditional model (without covariance adjustment)
ijkjkkjkkjktijk euTREATTREATy )()(0
),0(~ 2eijk Ne , ),0(~ 2
2Nu jk ,
22
23
0
0~
Nk
k
Given the variance component (VC) variance-covariance matrix
2 2 2 2 2 2 2 2 2 2 2 22 3 2 2 3
1 1 1 1*
2 4 2 4Total e e Total
ESV = 2 2 22 / Total
Recall that in the hierarchical design: 2Total = 2 2 21
4e .
Thus, we will have: 2 2 2 22 3 2
1
2 , i.e., 2 2 2 2 2
2 3
1*
2 Total
i.e., 22 3
1
2
MDES for matched-pair cluster-randomized design: without covariate:
JnJMMDES J
)1(424
*2
2
12
i.e.,
JnJ
MMDES J
)1(44 *3
*
12
JnJ
MMDES J
)1(4R14 *2B
*
12
, where *32
BR
.