Optimal Design for Longitudinal and Multilevel Research

Jessaca SpybrookJuly 10, 2008

*Joint work with Steve Raudenbush and Andres Martinez

Slide 2

Outline

Overview Examples

2-level HLM3-level HLM3-level Randomized Block Design

Writing up a Power Analysis Exercises

Overview

Tool for planning studies HLM notation Connection to Howard’s formulas

Warning: Optimal Design is easy to use but can also be easily misused

Slide 3

Connecting the Language

Larry OD2-level HLM Cluster Randomized Trial

3-level HLM Three level model with treatment at level three

3-level RBD Multi-site Cluster Randomized trial

Slide 4

Slide 5

Cluster Randomized Trial

(2-level HLM)

Three-Level Model with Treatment at Level 3

(3-level HLM)

Multisite Cluster Randomized Trial

(3-level RBD)

Number of Levels 2 3 3

Level of Randomization 2 3 2

Blocking No No Yes

Example of Level Structure

Students Schools

StudentsClassrooms

Schools

StudentsSchoolsDistricts

Slide 6

Organization of Optimal Design

Optimal Design for Group Randomized Trials Cluster Randomized Trial (2-level HLM) Three level model with treatment at level 3 (3-level

HLM) Multi-site cluster randomized trial (3-level RBD)

Cluster Randomized Trial (2-level HLM) ExampleProgram: A new math curriculum is randomly assigned and implemented at the school level.

Research question: Is the new math curriculum more effective than the traditional one?

Standardized math tests scores used as outcome measure. Based on previous research:

• 18% percent of the variation lies between schools.

(Students nested within schools).• Researchers expect the treatment to boost test scores by 0.25 standard deviations.

With 90 students per school, how many schools are needed to detect the above effect size with 80% power?

Slide 7

Slide 8

Cluster Randomized Trial (2-level HLM) Example

How many levels are in this study? What is the unit of randomization? Is there blocking? What is the MDES? What is a good estimate of the intraclass

correlation? What information is missing?

Cluster Randomized Trial (2-level HLM)

J – the total number of clusters n – the total number of individuals per cluster Rho – the intraclass correlation R-squared – the percent of variation explained

by level-2 covariate Delta – the effect size

Slide 9

Three level model with treatment at level three (3-level HLM) ExampleProgram: Third grade math curriculum implemented at the school level (group-level intervention). 40 schools are randomized. There are 6 teachers within each school and 25 kids per teacher.

Research question: Is the new math curriculum more effective than the traditional one?

Standardized math tests scores used as outcome measure. Based on previous research:

• 8% percent of the variation lies between classes. 12% of the variation lies between schools.

What is the MDES for power of 0.80? Is this reasonable?

Slide 10

Three level model with treatment at level three (3-level HLM) Example

How many levels are in this study? What is the unit of randomization? Is there blocking? What is the MDES? What are good estimates of the intraclass

correlations? What information is missing?

Slide 11

Three level model with treatment at level three (3-level HLM) K- the total number of schools J – the total number of classrooms per school n – the total number of individuals per classroom Rho pi (level-2 ICC)– the intraclass correlation at

classroom level Rho beta (level-3 ICC) – the intraclass correlation at

school level R-squared – the percent of variation explained by level-

3 covariate Delta – the effect size

Slide 12

Multi-site Cluster Randomized Trial (3-level RBD) ExampleProgram: The following will be repeated across districts. 6 schools will be randomly assigned to either the new math curriculum or the regular curriculum. 150 students will be tested in each school.

Causal question: Is the new math curriculum more effective than the traditional one?

Standardized math tests scores used as outcome measure. Based on previous research:

• 25% percent of the variation lies between districts.• We expect blocking will account for 30% of the variation.• Researchers expect the treatment to boost test scores by 0.20 standard deviations.

How many districts are needed to detect an effect size of 0.20 with 80% power?

Slide 13

Multi-site Cluster Randomized Trial (3-level RBD) Example How many levels are in this study? What is the unit of randomization? Is there blocking?

If so, how much of the variation is explained by blocking? How are blocks being treated – as fixed or random effects? If random effects, what is the variation of treatment effect across schools?

What is the MDES? What is the intraclass correlation prior to blocking? What information is missing?

Slide 14

Slide 15

Multi-site Cluster Randomized Trial (3-level RBD) K- the total number of blocks/sites (districts) J – the total number of schools per district n – the total number of individuals per school Rho – the intraclass correlation R-squared – the percent of variation explained by level-

3 covariate B – percent of variation explained by blocking Sigma sq sub delta – effect size variability (specifying

greater than 0 treats blocks as random effects) Delta – the effect size

Slide 16

Writing up a Power Analysis

Differs depending on the design (again it is critical to know your design!)

Slide 17

Two-level Cluster Randomized Trial

(2-level HLM)

Three-level Cluster Randomized Trial

(3-level HLM)

Three-level Multi-site Cluster Randomized

Trial(3-level RBD)

Number of level-one units

X X X

Number of level-two units

X X X

Number of level-three units

X X

Level 2 intra-class correlation

X X X

Level 3 intra-class correlation

X

Percent of variance explained by covariate

X X X

Percent of variance explained by blocking

X

Treatment of blocks – fixed versus random

X

Effect size variability X

Exercise 1

Suppose a group of researchers want to examine the effectiveness of a professional development program intended to improve reading instruction for elementary teachers. They plan to randomly assign schools to the treatment (the new pd).Within each school, they select four 2nd grade teachers to implement the program. Each teacher has approximately 25 students. The outcome of interest is student reading achievement. They expect that students in the treatment schools will improve their reading scores by 0.25 standard deviations. They plan to use last years students posttest as a covariate. How many schools are necessary for 80 percent power? (Hint: use the best estimates for the intraclass correlation(s))

Slide 18

Exercise 2

Suppose that a team of researchers plan to evaluate the effectiveness of a scripted whole school reform model. They plan to randomly assign entire schools to either receive the new reform model or continue with their current practices. On average, there are 300 students in each school. They expect that students in the treatment schools will outperform students in the control schools by 0.20 standard deviations on a particular math test. They estimate that approximately 20 percent of the variation lies between schools. They plan to use last years math scores as a covariate. How many schools do they need to have 80 percent power? What if they only tested 50 kids in the school, how does that affect the power?

Slide 19

Exercise 3

Suppose that a team of researchers recruited 6 districts for a study of the effectiveness of a new science program. They plan to randomly assign 4 schools within each district to either the new curriculum or their current curriculum and to test 150 students within each school. They expect that by blocking on district they will remove 25 percent of the variation in science outcome scores. They estimate an intraclass correlation of 0.24 and have a pre-test covariate that they think will explain approximately 60 percent of the variation in test scores. What is the smallest effect size they can detect with power of 0.80? Is this reasonable? (Hint: remember to go through the steps of selecting the appropriate design and all the relevant parts.)

Slide 20