Randomised Controlled Trials in the Social Sciences Analysis of randomised trials Martin Bland...

Randomised Controlled Trials in the Social Sciences

Analysis of randomised trials

Martin Bland

Professor of Health StatisticsUniversity of York

www-users.york.ac.uk/~mb55/

Trials in the social and health sciences

Randomisation began in agricultural research:

--- many treatments, complex designs,

--- research material plants,

--- few practical problems.

Randomisation in social and health sciences:

--- few treatments (usually 2), simple designs,

--- research material people, needing intervention,

--- many practical problems.


Practical problems:

must recruit in service setting, using service personnel,

must get consent, may have refusals unrepresentative samples,

misallocations in treatment may occur in service setting, due to mistakes, resource pressures, sabotage groups not being comparable,

treatments must be applied by service personnel, rather than by the researchers themselves, inconsistency,

subjects may drop out at any stage missing data,

may not be able to randomise individuals cluster randomisation.


Analytical problems:

non-comparable groups: intention to treat,

missing data: imputation,

allocation in groups: cluster level analyses, robust standard errors, multilevel modelling.

The Lanarkshire Milk Experiment: a warning from history

Nutritional experiment comparing ¾ pint milk per day with no milk, raw milk with pasteurised milk.

Spring 1931.

67 primary schools, 33 raw milk, 34 pasteurised.

Within schools, children allocated to “feeders” or “controls”.

20,000 children.

“Student” (W. G. Gossett) The Lanarkshire milk experiment. Biometrika 1931; 23: 398.


Allocation to “feeders” or “controls”

Left to the head teacher, groups to be “representative”.

“The teachers selected the two classes of pupils, those getting milk and those acting as controls, in two different ways. In certain cases they selected them by ballot and in others by an alphabetical system.” (Student, quoting original Report).

N.B. “classes” means “categories”, not school classes.



Left to the head teacher, groups to be “representative”.

“The teachers selected the two classes of pupils, those getting milk and those acting as controls, in two different ways. In certain cases they selected them by ballot and in others by an alphabetical system.” (Student, quoting original Report).

“So far so good, but after invoking the goddess of chance they unfortunately wavered in their adherence to her . . .” (Student).



“In any particular school where there was any group to which these methods had given an undue proportion of well-fed or ill-nourished children, others were substituted in order to obtain a more level selection.” (Student, quoting original Report.)

The controls were heavier and taller than the feeders.




“Presumably this discrimination in height and weight was not made deliberately, but it would seem probable that the teachers, swayed by the very human feeling that that the poorer children needed the milk more than the comparatively well to do, must have unconsciously made too large a substitution of the ill-nourished among the feeders and too few among the controls and that this unconscious selection affected, secondarily, both measurements.” (Student).


Measurement



Measurement

Children were weighed in their indoor clothes.

Start: February

End: June

“. . . since the selection was probably affected by poverty it is reasonable to suppose that the feeders would lose less weight from this cause than the controls.” (Student).


What should they have done?

Clearly, they should randomise and stick to the randomisation.

Once the trial had been done, could they retrieve it?

If they knew the original allocation, they could have analysed by intention to treat.

Analysis by intention to treat

We allocate subjects randomly so that we will have comparable groups which differ only in intervention and randomly.

Mistakes in treatment, sabotage, refusals, and drop-outs lead to non-comparable groups.

Solution: we analyse subjects in the comparable groups to which they were originally allocated

Analysis by intention to treat.


1954 field trial of Salk poliomyelitis vaccine: a lesson from history

Carried out using two different designs simultaneously, due to a dispute about the correct method.

In some districts, second grade school-children were invited to participate in the trial, and randomly allocated to receive vaccine or an inert saline injection (placebo).

In other districts, all second grade children were offered vaccination and the first and third grade left unvaccinated as controls.

Meier P. The biggest health experiment ever: the 1954 field trial of the Salk poliomyelitis vaccine. in Tanur JM et al. (eds.) Statistics: a Guide to the Biological and Health Sciences San Francisco: Holden-Day, 1977.

Analysis by intention to treat Result of the field trial of Salk poliomyelitis vaccine

Study group Number in Paralytic Polio group Number Rate per of cases 100000Randomized control: Vaccinated 200745 33 16 Control 201229 115 57 Not inoculated 338778 121 36

Observed control: Vaccinated 2nd grade 221998 38 17 Control 1st and 3rd grade 725173 330 46 Unvaccinated 2nd grade 123605 43 35

Poliomyelitis

Viral disease transmitted by the faecal-oral route.

Before the development of vaccine almost everyone in the population was exposed to it, usually in childhood.

In the majority of cases, paralysis does not result and immunity is conferred without the child being aware of having been exposed to polio.

In about one in 200 cases, paralysis or death occurs and a diagnosis of polio is made.

The older the exposed individual is, the greater the chance of paralysis developing.

Poliomyelitis

Children who are protected from infection by high standards of hygiene are likely to be older when they are first exposed to polio than those children from homes with low standards of hygiene, and thus more likely to develop the clinical disease.

There are many factors which may influence parents in their decision as to whether to volunteer or refuse their child for a vaccine trial.

These may include education, personal experience, current illness, and others, but certainly include interest in health and hygiene.

Thus in this trial the high risk children tended to be volunteered and the low risk children tended to be refused.

Poliomyelitis

The higher risk volunteer control children experienced 57 cases of polio per 100000, compared to 36/100000 among the lower risk refusers.

Suppose that the vaccine were saline instead, and that the randomised vaccinated children had the same polio experience as those receiving saline.




We would expect 200745 × 57 / 100000 = 114 cases, instead of the 33 observed.

Total cases in randomised areas would be 114 + 115 + 121 = 350 and the rate per 100000 would 47.


In the observed control areas of the Salk trial, the vaccinated and control groups are not comparable.

How could we analyse the trial?

Compare all second grade children, both vaccinated and refused, to the control group.




The rate in the second grade children

= (38 + 43) / ( 221998 + 123605) = 23 per 100,000.

Compare 46 per 100,000 1st and 3rd grade.


In the observed control areas of the Salk trial, the vaccinated and control groups are not comparable.

Compare all second grade children, both vaccinated and refused, to the control group.

The rate in the second grade children is 23 per 100,000, which is less than the rate of 46 in the control group, demonstrating the effectiveness of the vaccine.

The “treatment” which we are evaluating is not vaccination itself, but a policy of offering vaccination and treating those who accept.

This is analysis by intention to treat.


The random allocation procedure produces comparable groups and it is these we must compare, whatever selection may be made within them.

We therefore analyse the data according to the way we intended to treat subjects, not the way in which they were actually treated.

The alternative, analysing by treatment actually received, is called on treatment analysis or per protocol analysis.


Analysis by intention to treat is not free of bias.

As some participants may receive the other group's treatment, the difference may be smaller than it should be.

We know that there is a bias and we know that it will make the treatment difference smaller, by an unknown amount.

On treatment analyses are biased in favour of showing a difference, whether there is one or not.


Pasteurised or raw milk?

Some schools were provided with raw milk, some schools were provided with pasteurised milk.

The children in one school were allocated to the same type of milk.

“In so far as the conditions of this investigation are concerned the effects of raw and pasteurised milk on growth in weight and height are, so far as we can judge, equal.” (Fisher and Bartlett, 1931, quoting original Report).

Fisher RA, Bartlett S. Pasteurised and raw milk. Nature 1931; 127: 591-592.



“It is somewhat unfortunate . . . The whole of the milk supplied to any one school [was] either raw or pasteurised. In the absence of the records from the separate schools, it is impossible altogether to eliminate the doubt which this choice of method introduces.” (Fisher and Bartlett 1931).

The children in this trial were allocated as a group rather than as individuals.

It is a cluster allocated study.

(We do not know how clusters were allocated.)



“It is somewhat unfortunate . . . The whole of the milk supplied to any one school [was] either raw or pasteurised. In the absence of the records from the separate schools, it is impossible altogether to eliminate the doubt which this choice of method introduces.” (Fisher and Bartlett 1931).

Fisher RA, Bartlett S. Pasteurised and raw milk. Nature 1931; 127: 591-592.

Cluster randomised trials

Also called group randomised trials.

Research subjects are not sampled independently, but in a group.

For example:

all the patients in a general practice are allocated to the same intervention, the general practice forming a cluster,

all pupils in a school class are allocated to the same intervention, the class forming a cluster.


Members of a cluster will be more like one another than they are like members of other clusters.

We need to take this into account in the analysis and design.


Methods of analysis which ignore clustering:

two sample t method,

chi-squared test for a two way table,

difference between two proportions,

relative risk,

analysis of covariance,

logistic regression.

May mislead, because they assume that all subjects are independent observations.


Methods which ignore clustering may mislead, because they assume that all subjects are independent observations.

Observations within the same cluster are correlated.

May lead to standard errors which are too small, confidence intervals which are too narrow, P values which are too small.


A little simulation

Four cluster means, two in each group, from a Normal distribution with mean 10 and standard deviation 2.

Generated 10 members of each cluster by adding a random number from a Normal distribution with mean zero and standard deviation 1.

The null hypothesis, that there is no difference between the means in the two populations, is true.

Two-sample t test comparing the means, ignoring the clustering.


A little simulation

1000 times:

600 significant differences, with P<0.05

502 highly significant, with P<0.01.

If t test ignoring the clustering were valid, expect 50 significant differences, 5%, and 10 highly significant ones.

The analysis assumes that we have 20 independent observations in each group. This is not true.

We have two independent clusters of observations, but the observations in those clusters are really the same thing repeated ten times.


A little simulation: valid statistical analysis

Possible analysis:

• find the means for the four clusters

• carry out a two-sample t test using these four means only.

1000 simulation runs:

53 (5.3%) significant at P<0.05

14 (1.4%) highly significant at P<0.01


A little simulation

Simulation is very extreme.

Two groups of two clusters and a very large cluster effect.

Have seen a proposed study with two groups of two clusters.

Smaller cluster effect would only reduce the shrinking of the P values, it would not remove it.

Simulation shows that spurious significant differences can occur if we ignore the clustering.

How big is the effect of clustering?

The design effect is what we must multiply the sample size for a trial which is not clustered, to achieve the same power.

Alternatively, the power of a cluster randomised trial is the power of an individually randomised trial of size divided by the design effect.

Design effect:

Deff = 1 + (m − 1)×ICC

where m is the number of observations in a cluster and ICC is the intra-cluster correlation coefficient, the correlation between pairs of subjects chosen at random from the same cluster.


Deff = 1 + (m − 1)×ICC

If m =1, cluster size one, no clustering, then Deff =1, otherwise Deff will exceed 1.

ICC is usually quite small, 0.04 is a typical figure for health trials.

However, can be much larger.

In the incentives trial, Deff = 0.39.

Approximate design effect:

Deff = 1 + (6 − 1)×0.39 = 2.95


If we estimate the required sample size ignoring clustering, we must multiply it by the design effect to get the sample size required for the clustered sample.

Alternatively, if the sample size is estimated ignoring the clustering, the clustered sample has the same power as for a simple sample of size equal to what we get if we divide our sample size by the design effect.


Deff = 1 + (m − 1)×ICC

Clustering may have a large effect if the ICC is large OR if the cluster size is large.

E.g., if ICC = 0.001, cluster size = 500, the design effect will be 1 + (500 − 1)0.001 = 1.5,

Need to increase the sample size by 50% to achieve the same power as an unclustered trial.

Need to estimate variances both within and between clusters.

If the number of clusters is small, the between clusters variance will have few degrees of freedom and we will be using the t distribution in inference rather than the Normal. This too will cost in terms of power.

Analysis of cluster randomised trials

Several approaches can be used to allow for clustering:

summary statistics for each cluster

adjust standard errors using the design effect

robust variance estimates

general estimating equation models (GEEs)

multilevel modeling

Bayesian hierarchical models

others

Any method which takes into account the clustering will be a vast improvement compared to methods which do not.

A trial of incentives to attend adult literacy classes

Carole Torgerson, Greg Brooks, Jeremy Miles, David Torgerson

Classes randomised to incentive or no incentive.

Outcome variable: number of sessions attended.

Classes randomised to incentive or no incentive.

Two groups of 14 classes.

Labelled “X” and “Y” in this data set.

Blinded for analysis.

Group X: 77 students

Group Y: 86 students

0

5

10

15

20

25

Fre

qu

en

cy

0 5 10 15Sessions attended mid-post

Outcome variable: number of sessions attended.

Compare mean number of sessions ignoring clustering:

. ttest sessions , by(group) Two-sample t test with equal variances ------------------------------------------------------------------------------ Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]---------+-------------------------------------------------------------------- X | 70 6.685714 .4177941 3.495516 5.852238 7.519191 Y | 82 5.280488 .2991881 2.709263 4.685197 5.875778---------+--------------------------------------------------------------------combined | 152 5.927632 .2566817 3.164585 5.42048 6.434783---------+-------------------------------------------------------------------- diff | 1.405226 .5037841 .4097968 2.400656------------------------------------------------------------------------------Degrees of freedom: 150 Ho: mean(X) - mean(Y) = diff = 0 Ha: diff < 0 Ha: diff != 0 Ha: diff > 0 t = 2.7893 t = 2.7893 t = 2.7893 P < t = 0.9970 P > |t| = 0.0060 P > t = 0.0030



Stata version 8.



P = 0.006 — a highly significant difference!



P = 0.006 — a highly significant difference!

But it is wrong — it ignores the clustering!

Compare mean number of sessions ignoring clustering, regression:

. regress sessions group Source | SS df MS Number of obs = 152-------------+------------------------------ F( 1, 150) = 7.78 Model | 74.5694526 1 74.5694526 Prob > F = 0.0060 Residual | 1437.63449 150 9.58422997 R-squared = 0.0493-------------+------------------------------ Adj R-squared = 0.0430 Total | 1512.20395 151 10.0145957 Root MSE = 3.0958 ------------------------------------------------------------------------------ sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- group | -1.405226 .5037841 -2.79 0.006 -2.400656 -.4097968 _cons | 8.090941 .8152001 9.93 0.000 6.480183 9.701699------------------------------------------------------------------------------

P = 0.006 — identical to two sample t method.

It is still wrong — it ignores the clustering!

Compare mean number of sessions including clustering, two sample t method on cluster means:

. ttest sessions , by(group)

Two-sample t test with equal variances

------------------------------------------------------------------------------ Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]---------+-------------------------------------------------------------------- 1 | 14 6.69932 .7457716 2.790422 5.088178 8.310461 2 | 14 5.189229 .3974616 1.487165 4.330565 6.047893---------+--------------------------------------------------------------------combined | 28 5.944274 .439363 2.32489 5.042776 6.845773---------+-------------------------------------------------------------------- diff | 1.510091 .8450746 -.226985 3.247166------------------------------------------------------------------------------Degrees of freedom: 26

Ho: mean(1) - mean(2) = diff = 0

Ha: diff < 0 Ha: diff != 0 Ha: diff > 0 t = 1.7869 t = 1.7869 t = 1.7869 P < t = 0.9572 P > |t| = 0.0856 P > t = 0.0428

P = 0. 0856 — not significant.

Compare mean number of sessions including clustering, two sample t method on cluster means:

. ttest sessions , by(group)

Two-sample t test with equal variances

------------------------------------------------------------------------------ Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]---------+-------------------------------------------------------------------- 1 | 14 6.69932 .7457716 2.790422 5.088178 8.310461 2 | 14 5.189229 .3974616 1.487165 4.330565 6.047893---------+--------------------------------------------------------------------combined | 28 5.944274 .439363 2.32489 5.042776 6.845773---------+-------------------------------------------------------------------- diff | 1.510091 .8450746 -.226985 3.247166------------------------------------------------------------------------------Degrees of freedom: 26

Ho: mean(1) - mean(2) = diff = 0

Ha: diff < 0 Ha: diff != 0 Ha: diff > 0 t = 1.7869 t = 1.7869 t = 1.7869 P < t = 0.9572 P > |t| = 0.0856 P > t = 0.0428

P = 0. 0856 — not significant. Almost correct — it takes the data structure into account, but not the variation in class size.

Compare number of sessions including clustering, two sample t method on cluster means

Almost correct — it takes the data structure into account, but not the variation in class size.

02468

Fre

qu

en

cy

2 3 4 5 6 7 8 9 10Number in class

Group X

01234

Fre

qu

en

cy

2 3 4 5 6 7 8 9 10Number in class

Group Y

Compare mean number of sessions including clustering, regression method, weighted by class size:

. regress session group [aweight=learner](sum of wgt is 1.6300e+02) Source | SS df MS Number of obs = 28-------------+------------------------------ F( 1, 26) = 2.77 Model | 13.3075302 1 13.3075302 Prob > F = 0.1082 Residual | 124.992713 26 4.80741204 R-squared = 0.0962-------------+------------------------------ Adj R-squared = 0.0615 Total | 138.300243 27 5.12223123 Root MSE = 2.1926 ------------------------------------------------------------------------------ sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- group | -1.380902 .8299839 -1.66 0.108 -3.086958 .3251548 _cons | 8.00502 1.33388 6.00 0.000 5.26319 10.74685------------------------------------------------------------------------------

P = 0.108 — not significant.

Correct — it takes the data structure into account, including the variation in class size.

Compare individual number of sessions including clustering, robust standard error method (Huber-White-sandwich method):

. regress sessions group, cluster(class) Regression with robust standard errors Number of obs = 152 F( 1, 27) = 2.79 Prob > F = 0.1062 R-squared = 0.0493Number of clusters (class) = 28 Root MSE = 3.0958 ------------------------------------------------------------------------------ | Robust sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- group | -1.405226 .8407909 -1.67 0.106 -3.130387 .319934 _cons | 8.090941 1.535933 5.27 0.000 4.939466 11.24242------------------------------------------------------------------------------

P = 0.106 — not significant.

Correct — it takes the data structure into account.

Very similar estimate and P value to method using means.

Compare individual number of sessions including clustering, robust standard error method (Huber-White-sandwich method).



Compare individual number of sessions including clustering, robust standard error method (Huber-White-sandwich method.



I can do that using SPSS.

So what is the advantage?

Compare individual number of sessions including clustering, robust standard error method (Huber-White-sandwich method):



I can do that using SPSS.

So what is the advantage?

We can use subject-level covariates.

We can use subject-level covariates.

Mid-score = reading score before randomisation.

0

10

20

30

Fre

qu

en

cy

0 20 40 60 80 100Mid score (Scaled)

Compare individual number of sessions including clustering, robust standard error method, adjusting for mid-score:

. regress sessions group midscl, cluster(class) Regression with robust standard errors Number of obs = 152 F( 2, 27) = 11.91 Prob > F = 0.0002 R-squared = 0.1956Number of clusters (class) = 28 Root MSE = 2.8572 ------------------------------------------------------------------------------ | Robust sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- group | -1.533053 .6128085 -2.50 0.019 -2.790433 -.2756742 midscl | -.049151 .0104713 -4.69 0.000 -.0706363 -.0276658 _cons | 10.56678 1.304614 8.10 0.000 7.889936 13.24363------------------------------------------------------------------------------

P = 0.019 — significant.


Compare individual number of sessions including clustering, robust standard error method, adjusting for mid-score:

. regress sessions group midscl, cluster(class) Regression with robust standard errors Number of obs = 152 F( 2, 27) = 11.91 Prob > F = 0.0002 R-squared = 0.1956Number of clusters (class) = 28 Root MSE = 2.8572 ------------------------------------------------------------------------------ | Robust sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- group | -1.533053 .6128085 -2.50 0.019 -2.790433 -.2756742 midscl | -.049151 .0104713 -4.69 0.000 -.0706363 -.0276658 _cons | 10.56678 1.304614 8.10 0.000 7.889936 13.24363------------------------------------------------------------------------------

P = 0.019 — significant.


Adjustment produces true significant difference.

Randomised Controlled Trials in the Social Sciences

Analysis of randomised trials

Martin Bland

Professor of Health StatisticsUniversity of York

www-users.york.ac.uk/~mb55/

Randomised Controlled Trials in the Social Sciences Analysis of randomised trials Martin Bland...

Documents

Transcript of Randomised Controlled Trials in the Social Sciences Analysis of randomised trials Martin Bland...