Mixing different types of research in systematic reviews

Methodological Challenges for the Twenty First CenturyManchester, 22-23 November 2007

James Thomas

EPPI-Centre, Social Science Research Unit, Institute of Education, University of London

(2)

RESEARCH SYNTHESIS/ SYSTEMATIC REVIEWS

• A particular type or class of reviews

• Bridge between research and policy and practice

• Usually question-driven

• Use explicit methods, taking steps to increase trustworthiness

• Observational research or ‘Research on research’

(3)

Typical stages of a systematic review

Defining studies (inclusion and exclusion criteria)

Searching exhaustively (search strategy)

Describing the key features of studies

Assessing their quality/weight of evidence

Synthesising findings across studies

Setting question and developing protocol

Communication and engagement

(4)

Types of questions for systematic reviews*

• Effectiveness

• Screening and diagnosis

• Exploring risk or protective factors

• Observational associations between interventions and outcomes

• Questions about prevalence

• Questions about meanings and process

• Methodological questions

• Economic questions

* from Petticrew and Roberts (2005)

(5)

Examples of synthesis methods

• Statistical meta-analysis• Meta-ethnography• Grounded theory• Thematic analysis• Realist synthesis• Critical interpretive synthesis• Bayesian synthesis

Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A (2005) Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of

Health Services Research and Policy 10: 45-53.

(6)

Why synthesise quantitative research?

Meta-analysis refers to the analysis of analyses . . . the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research

literature.

Glass, 1976, p 3

(7)

From DiCenso et al. (2002) Interventions to reduce unintended pregnancies amongst adolescents: a systematic review of randomised controlled trials. British Medical Journal 231: 1426-1434

Does sex education increase the use of contraception amongst young people?

(8)

Why synthesise qualitative research?

“…the full contribution of qualitative research will not be realised if individual studies merely accumulate and some

kind of synthesis is not carried out…there are generalisations to be made across qualitative research

studies that do not supplant the detailed findings of individual studies, but add to them”

Britten et al. (2002)

(9)

Deriving Deriving descriptivedescriptive themesthemes

(10)

‘Mixed method’ systematic reviews (1/2)

• Policy and practice concerns often precede, or go beyond, questions of effectiveness

• Different types of questions likely to be answered by different types of findings

• Different types of findings may require different types of synthesis

(11)

‘Mixed method’ systematic reviews (1/2)

Single reviews with three syntheses

1) Effect sizes from trials pooled using statistical meta-analysis

2) Findings from qualitative studies synthesised using thematic analysis

3) Synthesis 2) used to interrogate synthesis 1)

(12)

An example of a ‘mixed method’ review

Children and healthy eating: a systematic review of barriers and facilitators*

*Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J (2003a) Children and Healthy Eating: A systematic review of

barriers and facilitators. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London (The full report of this review is

available at the EPPI-Centre website http://eppi.ioe.ac.uk/EPPIWeb/home.aspx)

:.

(13)

REVIEW PROCESS

Searching, screening and mapping

Synthesis 1: Trials (n=33)1. Quality assessment

2. Data extraction3. Statistical meta-analysis

Synthesis 2: Qualitative studies (n=8)

1. Quality assessment2. Data extraction

3. Thematic synthesis

Synthesis 3: Trials and qualitative studies

Focus narrowed to ‘fruit &veg’

(14)

Synthesis 1: Statistical meta-analysis

-0.5

0

0.5

1

1.5

2

2.5

Fruit and vegetables

Fruit only

Vegetables only

(15)

Line-by-line coding(Synthesis 2)

(16)

1) Children don’t see it as their role to be interested in health.

2) Children do not see future health consequences as personally relevant or credible.

3) Fruit, vegetables and confectionary have very different meanings for children.

4) Children actively seek ways to exercise their own choices with regard to foods.

5) Children value eating as a social occasion.

6) Children recognise contradiction between what is promoted and what is provided.

Children consider taste, not health, to be a key influence on their food choice

Food labelled as healthy may lead children to reject them (‘I don’t like it so it must be healthy’)

Buying healthy foods not seen as a legitimate use of their pocket money

Synthesis 2: Thematic analysis

(17)

Brand fruit and vegetables as ‘tasty’ rather than ‘healthy’.

Reduce health emphasis of messages

Do not promote fruit and vegetables in the same way within the same intervention.

Create situations for children to have ownership over their food choices.

Ensure messages promoting fruit and vegetables are supported by appropriate access to fruit and vegetables

1) Children don’t see it as their role to be interested in health.

2) Children do not see future health consequences as personally relevant or credible.

3) Fruit, vegetables and confectionary have very different meanings for children.

4) Children actively seek ways to exercise their own choices with regard to foods.

5) Children value eating as a social occasion.

6) Children recognise contradiction between what is promoted and what is provided.

Preparing for synthesis 3

(18)

Children’s views Trials

Recommendation for interventions

Good quality Other

Do not promote fruit and vegetables in the same way 0 0

Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one

5 5

Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health

5 6

Synthesis 3: Across studies

(19)

Increase (standardised portions per day) in vegetable intake across trials

0

0.1

0.2

0.3

0.4

0.5

0.6

War

dle

Liquo

ri

Henry

Ander

son

Reyno

lds Auld

Auld (b

)

Baran

owsk

i

Perry

Study

Po

rtio

ns

Little or no emphasis on health messages

Synthesis 3: Across studies

(20)

‘Added value’ of mixing methods

• Operationally, the method is simple, but conceptually, it is strong

• Integrates ‘quantitative’ estimates of benefit and harm with ‘qualitative’ understanding from people’s lives

• Facilitates a critical analysis of intervention studies from the point of view of those targeted by interventions - and vice versa

• Preserves the integrity of the findings of the different types of studies

(21)

James ThomasEPPI-Centre

SSRU18 Woburn Square

London, WC1H 0NR

Email: j.thomas@ioe.ac.uk

The methods described here can be found in: Thomas et al (2004) Integrating qualitative research with trials

in systematic reviews. British Medical Journal 328:1010-1012

For full details of the systematic review discussed in this paper and other EPPI-Centre reviews please see

our website:http://eppi.ioe.ac.uk/