Mixed methods synthesis

— ESRC Methods Festival 2006 —

James ThomasInstitute of Education, University of London

18 July 2006(2)

Background to ‘mixed methods’ approach

• Policy and practice concerns often precede, or go beyond, questions of effectiveness.

• Different types of questions require different combinations of study types to be included.

• Different combinations of study types demand different methods of synthesis.

• However, key principles of systematic reviews are not compromised.

18 July 2006(3)

Case example of a ‘mixed methods’ synthesis

What is known about the barriers to, and facilitators of, healthy eating amongst

children?*

*The full report of this review is available at the EPPI-Centre website: Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J (2003a) Children and Healthy Eating: A systematic review of barriers and facilitators. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

18 July 2006(4)

Review questione.g. What is known about the barriers to, and facilitators of, fruit and

vegetable intake amongst children aged 4 to 10 years?

Trials (N=33) 1. Application of inclusion criteria

2. Quality assessment3. Data extraction

4. Synthesis using statistical meta-analysis

‘Views’ studies (N=8) 1. Application of inclusion criteria

2. Quality assessment3. Data extraction

4. Thematic synthesis

Trials and ‘views’Mixed methods synthesis

SCOPING AND MAPPING (of 272 reports of 193 studies)

Review process

18 July 2006(5)

One review with three syntheses

1. ‘Quantitative’ methods used to conduct a meta-analysis of data from trials.

2. ‘Qualitative’ methods used to synthesise textual data from ‘views’ studies (aided by NVivo).

3. Both ‘qualitative’ and ‘quantitative’ methods used to combine experimental studies of effectiveness (trials) with studies of people’s views (descriptive).

18 July 2006(6)

Quality-assessment methods

• SYNTHESIS 1• Quantitative (Trials)• Provision of pre- and post-

data on outcomes• Provision of data on all

outcomes measured• Employment of equivalent

control/comparison group• Resulted in ‘high’,

‘medium’ and ‘not sound’/ ‘low’ trials

• SYNTHESIS 2• Qualitative (‘Views’)• Quality of reporting (5

items)• Sufficiency of strategies

for reliability/validity (4 items)

• Extent to which study findings were rooted in children’s own perspectives (3 items)

18 July 2006(7)

Methods for synthesis 1: ‘Quantitative’ (Trials)

• Effect sizes from trials pooled using:– Statistical meta-analysis – Six different outcomes

• Heterogeneity across studies explored via:– Sub-group analysis– Qualitative analysis of textual data from trials

18 July 2006(8)

Findings for synthesis 1: ‘Quantitative’ (Trials)

18 July 2006(9)

Methods for synthesis 2: ‘Qualitative’ (Views studies)

Stage 1: Thematic analysis of textual data (study authors’ descriptions of findings)

= Descriptive themes

Stage 2: Descriptive themes examined in light of review question

= Barriers, facilitators and recommendations for interventions

Stage 3: Thematic analysis revised

= Analytical themes

18 July 2006(10)

Findings of synthesis 2: Descriptive themes

Influences on

foods eaten

Chosen foods

Provided foods

Foods in the home

Foods in the school

Parental influence and food rules Breaking rules

Limited choiceSocial occasionContradictions

Food preferencesPerceptions of health benefitsKnowledge-behaviour gapRoles and responsibilitiesNon-influencing factors

Concepts of healthy eating

Healthy eating conceptsGood and bad foodsHealth consequences

18 July 2006(11)

Findings of synthesis 2: Analytical themes, barriers, facilitators, and interventions implications

Brand fruit and vegetables as ‘tasty’ rather than ‘healthy’.

Reduce health emphasis of messages

Do not promote fruit and vegetables in the same way within the same intervention.

Create situations for children to have ownership over their food choices.

Ensure messages promoting fruit and vegetables are supported by appropriate access to fruit and vegetables

1) Children don’t see it as their role to be interested in health.

2) Children do not see future health consequences as personally relevant or credible.

3) Fruit, vegetables and confectionary have very different meanings for children.

4) Children actively seek ways to exercise their own choices with regard to foods.

5) Children value eating as a social occasion.

6) Children recognise contradiction between what is promoted and what is provided.

18 July 2006(12)

METHODS FOR SYNTHESIS 3: ACROSS STUDY TYPES

• Product of thematic synthesis of views studies was the mechanism to combine trials and views studies.

• Conceptual and methodological matrix juxtaposed findings from views synthesis against findings from trial synthesis.

• Comparative analysis guided by 3 questions:

– Which interventions match recommendations derived from children’s views and experiences?

– Which recommendations have yet to be addressed by soundly evaluated interventions?

– Do those interventions which match recommendations show bigger effect sizes and/or explain heterogeneity?

18 July 2006(13)

Synthesis 3 (Across Study Types)

Children’s Views Trials

Recommendation for interventions

Good quality Other

Do not promote fruit and vegetables in the same way

None None

Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one

5 5

Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health

5 6

18 July 2006(14)

Synthesis 3: Sub-group Analysis

Increase (standardised portions per day) in vegetable intake across trials

0

0.1

0.2

0.3

0.4

0.5

0.6

War

dle

Liquo

ri

Henry

Ander

son

Reyno

lds Auld

Auld (b

)

Baran

owsk

i

Perry

Study

Po

rtio

ns

Little or no emphasis on health messages

18 July 2006(15)

CONCLUSIONS

• Conceptually, our method allows us to integrate ‘quantitative’ estimates of benefit and harm with ‘qualitative’ understanding from people’s lives.

• Technically, the insights gained from the

qualitative synthesis of ‘views’ studies allows the exploration of heterogeneity in ways in which it would be difficult to imagine in advance.

• Our approach has raised a number of methodological and conceptual challenges…

18 July 2006(16)

CHALLENGES

• To notions of defining sub-group analyses a priori.

• To public health interventions developed and trialled solely by ‘experts’.

• To traditional polarisation of qualitative and quantitative research (bridging paradigms).