11

Assessing quality in systematic Assessing quality in systematic reviews of the effectiveness of reviews of the effectiveness of health promotion and public health promotion and public

health (HP/PH): Areas of health (HP/PH): Areas of consensus and dissensionconsensus and dissension

Dr Jonathan ShepherdDr Jonathan ShepherdPrincipal Research FellowPrincipal Research Fellow

Southampton Health Technology Assessments Centre Southampton Health Technology Assessments Centre (SHTAC),(SHTAC),

School of Medicine,School of Medicine,University of Southampton, UKUniversity of Southampton, UK

22

Objectives of today’s Objectives of today’s presentationpresentation

Present results of a series of interviews Present results of a series of interviews with systematic reviewers in HP/PHwith systematic reviewers in HP/PH Challenges faced by systematic reviewers Challenges faced by systematic reviewers Barriers / facilitators around assessing qualityBarriers / facilitators around assessing quality Barriers / facilitators around learning to do Barriers / facilitators around learning to do

systematic reviews of HP/PHsystematic reviews of HP/PH

33

MethodsMethods

Stage 1 – detailed methodological Stage 1 – detailed methodological mapping of random sample of HP/PH mapping of random sample of HP/PH systematic reviews (n=30)systematic reviews (n=30)

Stage 2 – Semi-structured interviews with Stage 2 – Semi-structured interviews with a sample of systematic reviewers in a sample of systematic reviewers in HP/PH (n=17)HP/PH (n=17)

44

Methods (2)Methods (2)

Purposive (n=17)Purposive (n=17) Cochrane Colloquium (Melbourne, 2005) Cochrane Colloquium (Melbourne, 2005)

(n=10/17 interviews)(n=10/17 interviews) UK / telephone (2006) (n=7/17 interviews)UK / telephone (2006) (n=7/17 interviews) Content analysis approach to categorise Content analysis approach to categorise

data into themesdata into themes Data analysed using NVivo (2.0)Data analysed using NVivo (2.0)

55

Methods (3)Methods (3): Interviewees: Interviewees

Interviewee’s countryInterviewee’s country Number (%) interviewedNumber (%) interviewed

AustraliaAustralia 4 (26)4 (26)

CanadaCanada 2 (12)2 (12)

NigeriaNigeria 1 (6)1 (6)

South AfricaSouth Africa 1 (6)1 (6)

United KingdomUnited Kingdom 8 (47)8 (47)

USAUSA 1 (6)1 (6)

66

Results (1): challenges to doing systematic reviewsResults (1): challenges to doing systematic reviews CategoryCategory ChallengeChallenge Number (%) ofNumber (%) of

intervieweesinterviewees

External factorsExternal factors Lack of resources and timeLack of resources and time 11 (65)11 (65)

Analysis and methodsAnalysis and methods Analysing complex interventionsAnalysing complex interventions 5 (29)5 (29)

Locating the evidence Locating the evidence 4 (24)4 (24)

Analysing observational studies Analysing observational studies 3 (18)3 (18)

Critical appraisal Critical appraisal 3 (18)3 (18)

Dealing with heterogeneity and statisticsDealing with heterogeneity and statistics 3 (18)3 (18)

Developing the review questionDeveloping the review question 2 (12)2 (12)

Lack of flexibilityLack of flexibility 1 (6)1 (6)

Evidence limitationsEvidence limitations Reporting biasReporting bias 2 (12)2 (12)

Poor evidencePoor evidence 1 (6)1 (6)

ExpertiseExpertise Team working Team working 1 (6)1 (6)

Lack of specialist knowledgeLack of specialist knowledge 1 (6)1 (6)

MiscellaneousMiscellaneous Lack of motivation + tedium Lack of motivation + tedium 1 (6)1 (6)

Advisory group Advisory group 1 (6)1 (6)

Interpretation and presentationInterpretation and presentation 1 (6)1 (6)

Supporting external groups Supporting external groups 1 (6)1 (6)

77

Results (2): barriers to, and Results (2): barriers to, and facilitators of, quality assessmentfacilitators of, quality assessment

Facilitating factorsFacilitating factors Number (%) Number (%) intervieweesinterviewees

Well written-up studiesWell written-up studies 9 (53)9 (53)

Good critical appraisal criteria / Good critical appraisal criteria / instrumentinstrument

9 (53)9 (53)

RCTs / experimental studiesRCTs / experimental studies 3 (18)3 (18)

Good guidelinesGood guidelines 2 (12)2 (12)

SupportSupport 2 (12)2 (12)

Consensus on criteriaConsensus on criteria 1 (6)1 (6)

88

Results (3): Barriers to, and facilitators Results (3): Barriers to, and facilitators of, quality assessment (cont)of, quality assessment (cont)

““I mean it is difficult in, you know that there are judgement I mean it is difficult in, you know that there are judgement calls all the time, it isn’t you know it, none of these things calls all the time, it isn’t you know it, none of these things are nice easy tick boxes and there’s so many judgements are nice easy tick boxes and there’s so many judgements

right the way through”right the way through”

““I’ve never met a set of guidelines where they I’ve never met a set of guidelines where they completely dispense with the idea that you make a new completely dispense with the idea that you make a new

judgement”judgement”

Interviewee 14Interviewee 14

99

Results (4): Consensus on quality Results (4): Consensus on quality assessment criteria?assessment criteria?

Majority thought there was generally no Majority thought there was generally no consensus…consensus…

……but more consensus for RCTsbut more consensus for RCTs Concept of “Multiple consensus”Concept of “Multiple consensus” Realistic to achieve consensus in HP/PH?Realistic to achieve consensus in HP/PH?

1010

Results (4) consensus on quality Results (4) consensus on quality assessment criteria (cont.)assessment criteria (cont.)

““I’m not sure that that matters that much because I’m not sure that that matters that much because people will always do things in different ways. And people will always do things in different ways. And the idea of forcing an orthodoxy in an area which is the idea of forcing an orthodoxy in an area which is

incredibly diverse as public health systematic incredibly diverse as public health systematic reviews. I mean, I think imagination and lateral reviews. I mean, I think imagination and lateral

thinking bring just as much other help as thinking bring just as much other help as

systematic methods”systematic methods”

Interviewee 13Interviewee 13

1111

Results (5) Learning to do Results (5) Learning to do systematic reviewssystematic reviews

Method of learningMethod of learning Number (%) of Number (%) of intervieweesinterviewees

‘‘Hands-on’ practice Hands-on’ practice 11 (65)11 (65)

From colleagues and mentorsFrom colleagues and mentors 11 (65)11 (65)

Training coursesTraining courses 10 (59)10 (59)

Literature and written resourcesLiterature and written resources 9 (53)9 (53)

Applied existing research skillsApplied existing research skills 5 (29)5 (29)

Academic courseAcademic course 3 (18)3 (18)

Supervising and teaching othersSupervising and teaching others 2 (12)2 (12)

1212

Results (6): Challenges in helping Results (6): Challenges in helping others to do reviewsothers to do reviews

IssuesIssues Number (%) of Number (%) of intervieweesinterviewees

Meta-analysis Meta-analysis 7 (54)7 (54)

Quality assessmentQuality assessment• The need to think criticallyThe need to think critically• Using judgmentUsing judgment• Conceptually challengingConceptually challenging

4 (31)4 (31)

Principles of evidence-based healthPrinciples of evidence-based health 2 (15)2 (15)

Reading and synthesising evidenceReading and synthesising evidence 2 (15)2 (15)

Study designs and hierarchy of Study designs and hierarchy of evidenceevidence

1 (8)1 (8)

Qualitative dataQualitative data 1 (8)1 (8)

1313

Conclusions: key overarching Conclusions: key overarching themesthemes

1.1. Complexity of many HP/PH reviewsComplexity of many HP/PH reviews

1414

““It’s big and diverse, that’s why it’s so much easy It’s big and diverse, that’s why it’s so much easy just to stick with randomised control trials and be just to stick with randomised control trials and be done with it. Or even if you do community trials, done with it. Or even if you do community trials, cluster randomised, still it’s easier. I think once cluster randomised, still it’s easier. I think once

you move beyond it gets very difficult”you move beyond it gets very difficult”

Interviewee 4 Interviewee 4

Conclusions: key overarching Conclusions: key overarching themes (cont)themes (cont)

1515

Conclusions: key overarching Conclusions: key overarching themes (cont)themes (cont)

1.1. Complexity of many HP/PH reviewsComplexity of many HP/PH reviews

2.2. Time and resourcesTime and resources

3.3. SubjectivitySubjectivity

1616

RecommendationsRecommendations

Adequate planning and resourcing of Adequate planning and resourcing of complex HP/PH systematic reviewscomplex HP/PH systematic reviews

Better reporting of HP/PH studiesBetter reporting of HP/PH studies Training for critical appraisal Training for critical appraisal

opportunities for ‘hands on’ learningopportunities for ‘hands on’ learning acknowledging subjectivityacknowledging subjectivity

1717

AcknowledgementsAcknowledgements

University of SouthamptonUniversity of Southampton Professor Katherine WeareProfessor Katherine Weare Interviewees who took part in the researchInterviewees who took part in the research EPPI-Centre, LondonEPPI-Centre, London Cochrane Public Health GroupCochrane Public Health Group

email: jps@soton.ac.ukemail: jps@soton.ac.uk