1

Repeated-measures data in educational research trials – how should it be analysed?

Ben Styles

Senior Statistician

National Foundation for Educational Research

2

3

Two sweeps example

• Cluster randomised trial of reading materials

• Baseline reading test, 10 week intervention, follow-up reading test

• Two parallel versions of the Suffolk Reading Scale

4

Using baseline data as a covariate in a multi-level (pupil, school) regression model

Different analysis, different results

Outcome Background Coefficient SE p

Post-test score Constant 10.95 0.6876 0.000 ***

Intervention 0.672 0.5742 0.242 NS

Pre-test score 0.7708 0.0167 0.000 ***

Different analysis, different results

Using time as a level in a repeated measures multi-level (time, pupil, school) regression model

5

Outcome Background Coefficient SE p

Total score Constant 32.35 1.206 0.000 ***

Time 3.387 0.3593 0.000 ***

Intervention -1.682 1.684 0.318 NS

Time*intervention 1.238 0.4916 0.012 *

Interaction

6

Effect of RAPID

25

27

29

31

33

35

37

39

TestA TestB

Raw

tes

t sc

ore

Control

Intervention

Six sweeps example

• Mentoring scheme for struggling readers• Pupil-level randomisation• Questionnaire administered once at

baseline and then every four months for the next two years

7

Six sweeps example

Using time as a level in a repeated-measures multi-level (time, pupil, school) regression model

8

Outcome Background Coefficient SE pAspirations for the future Constant 22.65 0.131 0.000***

Time -0.07358 0.02625 0.005**

Intervention -0.2796 0.1694 0.099NS

Time*intervention 0.08131 0.03801 0.032*

Reading

• Two-waves studies cannot describe individual trajectories of change and they confound true change with measurement error (Singer and Willett, 2002)

• ANCOVA is valid even with pre-test measurement error (Senn, 2004)

• Unconditional change models described in text books have three or more time-points

• The ANCOVA will almost always provide a more powerful test of the hypothesis of interest than will the repeated measures ANOVA approach (Dugard and Todman, 1995)

9

Change model assumption violation (2 sweeps)

10

Change model assumption OK (six sweeps)

11

Correlations r1 r2 r3 r4 r5 r6r1 Pearson Correlation 1.000 -0.002 -0.002 0.023 -0.002 -0.003

Sig. (2-tailed) 0.000 0.957 0.957 0.549 0.969 0.957N 843 680 675 674 656 347

r2 Pearson Correlation -0.002 1.000 -0.002 0.016 0.121 -0.039Sig. (2-tailed) 0.957 0.000 0.968 0.700 0.003 0.490N 680 680 610 606 591 315

r3 Pearson Correlation -0.002 -0.002 1.000 0.047 -0.002 -0.003Sig. (2-tailed) 0.957 0.968 0.000 0.237 0.968 0.955N 675 610 675 622 603 316

r4 Pearson Correlation 0.023 0.016 0.047 1.000 -0.019 0.065Sig. (2-tailed) 0.549 0.700 0.237 0.000 0.636 0.237N 674 606 622 674 612 331

r5 Pearson Correlation -0.002 0.121 -0.002 -0.019 1.000 -0.003Sig. (2-tailed) 0.969 0.003 0.968 0.636 0.000 0.956N 656 591 603 612 656 321

r6 Pearson Correlation -0.003 -0.039 -0.003 0.065 -0.003 1.000Sig. (2-tailed) 0.957 0.490 0.955 0.237 0.956 0.000N 347 315 316 331 321 347

Measurement error problematic

12

Measurement error problematic

13

Measurement error not a problem

14

A better repeated measures model

15

A (slightly) better conditional model

16

Conclusion

• No consensus but it is probably safer to use a conditional model for a pre-test post-test design

• Designs with three or more sweeps will benefit from a repeated measures multi-level model

• Care with level 1 residual autocorrelation• Try a few models and check assumptions• Don’t get hung up on significance

17

Questions and advice

18

Acknowlegements

Pearson

Business in the Community and Queen’s University, Belfast

Tom Benton

Dougal Hutchison

NFER

19