www.joannabriggs.edu.au

Systematic Reviews: the process, quantitative, qualitative and mixed

methods reviews

Edoardo Aromataris

www.joannabriggs.edu.au

Systematic reviews• Different strokes...

– Where I’m coming from• Introduction to reviews and systematic reviews

– Terminology• Systematic steps• Journey beyond where you may be involved...• Where you’ll find differences in quantitative and

qualitative

www.joannabriggs.edu.au

Some answers!?• What is a systematic review?

www.joannabriggs.edu.au

Some answers!?• An attempt to identify, appraise and synthesise all

the empirical evidence that meets pre-specified eligibility criteria to answer a given research question. 

• An attempt to sum up the best available research on a specific question. This is done by synthesising the results of several studies.

www.joannabriggs.edu.au

• “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, updated periodically, of all relevant randomized controlled trials” Archie Cochrane, 1972

The Nature of Evidence

www.joannabriggs.edu.au

Cochrane Collaboration• Established 1993• Leaders in EBM, some 28,000 members• Focus on effectiveness of interventions and

therapies in medical practice– RCTs focus– More pluralistic – qualitative methods

• Organised into Cochrane Review Groups (CRG)• Cochrane Library

www.joannabriggs.edu.au

Campbell Collaboration (C2)• Focus on systematic reviews of effectiveness in

education, crime and justice, and social welfare• Coordinating groups (6)• Campbell Systematic Reviews

• Synthetic Reviews

www.joannabriggs.edu.au

• “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, updated periodically, of all relevant randomized controlled trials” Archie Cochrane, 1972.

• Shift toward pluralistic, inclusive definitions of what evidence is, and subsequently of what evidence based practice is...

The Nature of Evidence

www.joannabriggs.edu.au

Joanna Briggs Institute• Established 1996• Feasibility, Appropriateness, Meaningfulness and

Effectiveness• Qualitative methods• JBC – 70+ Centres conduct reviews• JBI Library of Systematic Reviews• Leaders in EBHC and evidence implementation

www.joannabriggs.edu.au

Introduction to Systematic Reviews• Variations in conduct• Different questions, different methods of synthesis

– E.g. effectiveness, aetiology, harms, meta-ethnographic reviews, realist syntheses

• Not all are validated– Term used loosely

• Broad coverage, applicable to most reviews– Systematic!

www.joannabriggs.edu.au

Terminology• Literature review summarises, critiques, and

synthesizes articles while not using systematic methodology

• Systematic reviews adhere to explicit and rigorous methods to identify, critically appraise, and synthesise relevant primary/original studies.

(Krainovich-Miller, 2006:87)

www.joannabriggs.edu.au

Systematic Review• Also called “Research Synthesis”• Is an attempt to integrate empirical data for the

purpose of:– uncovering the international evidence and– producing statements about that evidence to guide

decision making• Requires explicit and exhaustive reporting of the

methods used in synthesis

www.joannabriggs.edu.au

Systematic Review

• ‘…an attempt to minimize the element of arbitrariness…by making explicit the review process, so that, in principle, another reviewer with access to the same resources could undertake the review and reach broadly the same conclusions’

(Dixon-Wood et al. 1997:157 quoted by Seers, 2005:102)

www.joannabriggs.edu.au

Bias

• ‘systematic deviation of results or inferences from truth’

• ‘an error in the conception and design of a study-or in the collection, analysis, interpretation, reporting, publication, or review of data-leading to results or conclusions that are systematically (as opposed to randomly) different from truth’ (Porta, 2008:18)

www.joannabriggs.edu.au

Meta-analysis • Quantitative evidence

– Questions of Effectiveness, Feasibility and/or Appropriateness

• Use of statistical methods to combine the results of various independent, similar studies

• More precise calculation of one estimate of treatment effect than could be achieved by any of the individual, contributing studies

• Only forms a part of the systematic review in which it appears

www.joannabriggs.edu.au

Meta-synthesis

• Qualitative evidence– Questions of Meaningfulness, Feasibility and/or Appropriateness

• Qualitative analysis of a number of independent qualitative research studies and text

• Use of qualitative methods of combining the findings of individual studies

• Only forms a part of the systematic review in which it appears

www.joannabriggs.edu.au

Qualitative Research Findings as Evidence for Practice• Qualitative evidence is of increasing importance in health

services policy, planning and delivery.• It can play a significant role in:

– understanding how individuals / communities perceive health, manage their own health and make decisions related to health service usage;

– increasing our understandings of the culture of communities and of health units;

– Informing planners and providers;– evaluating components and activities of health services that

cannot be measured in quantitative outcomes.

www.joannabriggs.edu.au

Systematic Review• The notion of and methods for establishing credibility in

systematic reviews has been extensively developed and debated

• In terms of quantitative evidence:– Emphasis on reducing bias and increasing validity– Degree of credibility established through critique and by applying

levels of evidence• In terms of qualitative evidence:

– Emphasis on rigour of research design and transferability– Degree of credibility established through critique and by applying

levels of credibility

www.joannabriggs.edu.au

Comprehensive/Mixed method Review• Series of questions• Combines both quantitative and qualitative findings

and addresses multiple forms of evidence• 2 or more types of evidence• Different approaches - particularly to integration of

qualitative findings

www.joannabriggs.edu.au

Characteristics of a SR• Protocol driven process• Clearly stated set of objectives with pre-defined eligibility

criteria for studies• Explicit, reproducible methodology • Systematic search that attempts to identify all studies that

would meet the eligibility criteria• Assessment of the validity of the findings of the included

studies • Systematic presentation, and synthesis, of the characteristics

and findings of the included studies(Green et al., 2008:6)

www.joannabriggs.edu.au

Steps in a Systematic Review• Formulate review question• Define inclusion and exclusion criteria• Locate studies• Select studies• Assess study quality• Extract data• Analysis/summary and synthesis of relevant studies• Present results• Interpret results/determining the applicability of results

(Egger & Davey Smith, 2001:25; Glasziou et al., 2004:2)

www.joannabriggs.edu.au

Question Development

• Reviews of effects & economics:– Population– Intervention/Exposure– Comparator– Outcome– Types of Study Design

• Reviews of qualitative & Textual data:– Population– Phenomena of Interest– Context

Divide question following the PICO/PICo model

www.joannabriggs.edu.au

Question?• Very important - guides entire review• Effectiveness? Meaningfulness?• Other components - How? Why? When?

www.joannabriggs.edu.au

Where to Start?• Conduct a search for previous Systematic

Reviews on your topic: has this review already been conducted or is a review already in progress? – Yes = stop and refine your question– No = proceed with your protocol

development

www.joannabriggs.edu.au

Protocol• Detailed review methods a priori• Guide process

– Reasoned approach to the question asked• Decrease biased post hoc changes

– Important to avoid “fishing”• Review proper may deviate

– Needs clear explanation how and why

www.joannabriggs.edu.au

Protocol Development• The usefulness and success of the review stems

from the robustness of the protocol• The protocol:

– Guides the specific direction of the review– Describes inclusion criteria– Identifies the appropriate search sources and

resources– Methods of appraisal, extraction and synthesis

www.joannabriggs.edu.au

Background• Describe the issue under review, including:

– Target population, interventions, outcomes, • Should concisely overview the main elements of the

review, and issues within the topic of choice• Provide adequate detail to justify the conduct of the

review and choice of inclusion criteria• Provide necessary definitions of important terms

and concepts

www.joannabriggs.edu.au

Inclusion Criteria: Quantitative data

• Study designs • Population characteristics• Intervention or exposure• Comparator - active or passive• Outcomes of interest

www.joannabriggs.edu.au

Inclusion Criteria: Qualitative & Text

• Study designs • Population characteristics• Phenomena of Interest• Context

www.joannabriggs.edu.au

Search Strategy

• Know and understand the question!• What is the relevant PICO / PICo?• Qualitative, quantitative or economic?• Feasibility, Appropriateness, Meaningfulness or

Effectiveness?• Other questions? Extra concepts? Less

concepts?

www.joannabriggs.edu.au

Search Strategy Steps• Initial Search

– initial search of MEDLINE, CINAHL, followed by analysis of text words in the title and abstract

• Second Search– all identified key words and index terms across all

databases• Third Search

– references of identified studies and unpublished studies

www.joannabriggs.edu.au

Search Strategy• Features of search strategy

– Sensitivity – ability to identify all the relevant studies– Specificity – ability to exclude irrelevant studies

• Inverse relationship between sensitivity and specificity– High sensitivity will tend to have low specificity– A large number of articles retrieved may not be relevant

to the review question

www.joannabriggs.edu.au

Searching• Depend on methods (to minimise bias, or not?)

– Exhaustive or Saturation based?– Limited or comprehensive?

• Impact on transparency and auditability• Where appropriate?

– How much is enough?– Are appropriate sources being covered?

www.joannabriggs.edu.au

Types of Sources

• What to use?– Scientific databases – Scientific Journals– Organisations– Websites– Libraries– Experts

www.joannabriggs.edu.au

Types of Resources• Peer reviewed journal articles

– Research– Opinion/commentary/letters

• Grey Literature– Google Scholar, Mednar

• Theses/Dissertations– DATAD

• Data – Statistics• Circulars• Reports

www.joannabriggs.edu.au

Databases and sources…• Medline/PubMed• Scopus/Embase• POPLINE• CINAHL• ERIC• PsychINFO

Consult your Librarian!

• ScienceDirect• TRIP• Wiley InterScience• SPORTDiscus• ISI Web of Science• + many more…!

www.joannabriggs.edu.au

Research and Trials Registers• Cochrane library

– CCTR• Clinical trials.gov• Controlled Clinical Trials• NHS Research Register• REGARD (database of the ESRC)

www.joannabriggs.edu.au

Grey literature

• Mednar• PsycEXTRA • OAIster • SIGLE• Google Scholar

www.joannabriggs.edu.au

Theses & Dissertations• Index to Thesis • Networked Digital Library of Theses and

Dissertations (NDLTD)• ProQuest Dissertations and Theses Database• EThOS - Beta - Electronic Thesis Online System• TROVE (ADTP)• +…

www.joannabriggs.edu.au

Government Websites• Are there government agencies that may have

evidence relevant to your review question?– NCCAM– CDC– NH&MRC– AHRQ– AIHW– +…

Search Logistics!

• Apply Search Strategy to databases

• Export to bibliographic software– E.g. Endnote

• Document process

Search Strategy

Search Sources

Export citationsto bibliographic software

www.joannabriggs.edu.au

Study Selection• Study Selection is an initial assessment that occurs

following the review Search• It addresses the question “should the paper be

retrieved?”– 2nd assessment that occurs after retrieval and addresses the

question “should the study be included in the review?” - this is CRITICAL APPRAISAL!

• It is essential to use two assessors in both the selection and critical appraisal processes to limit the risk of error

www.joannabriggs.edu.au

Selection Process• Aims to select only those studies that address the

review question and that match the inclusion criteria documented in your protocol

• Scan titles and abstracts• Err on the side of caution - Inclusive!• If uncertain? - Retrieve - scan full text• The selection should be:

– Transparent– Reproducible

www.joannabriggs.edu.au

Critical Appraisal• Aim is to establish validity

– To establish the risk of bias– Internal validity

• Every review must set out to use an explicit appraisal process. Essentially,– A good understanding of research design is required in

appraisers; and– The use of an agreed checklist or scale is usual.

Why Critically Appraise?

• Combining results of poor quality research may lead to biased or misleading results and understandings

1004 references

832 referencesScanned Ti/Ab

172 duplicates

117 studiesretrieved

715 do not meetIncl. criteria

82 do not meetIncl. criteria

35 studies forCritical Appraisal

www.joannabriggs.edu.au

Assessing the Risk of Bias• Numerous tools are available for assessing methodological

quality of clinical trials and observational studies.• Generally requires the use of a specific tool for assessing risk

of bias in each included study. • ‘High quality’ research methods can still leave a study at

important risk of bias (e.g. when blinding is impossible)• Some markers of quality are unlikely to have direct

implications for risk of bias (e.g ethical approval, sample size calculation)

Type of bias Quality assessment

Population

Allocation

Selection Allocation concealment

Treatment Control

Performance Blinding Exposed to intervention

Not exposed

Detection Blinding Population Population

Attrition ITT follow up Follow up Follow up

JBI-MAStARI Instrument

www.joannabriggs.edu.au

Critical Appraisal• Exclude studies on basis of appraisal?• Explore impact through sensitivity

analyses/subgroups

www.joannabriggs.edu.au

Critical Appraisal & Qualitative Review• General acceptance of need for quality• Ongoing debate around the role of appraisal• Particular focus of debate on role of scales and sum

scores in appraisal• In practice, appraisal instruments for qualitative

research tend to focus on establishing the degree to which the evidence applies to practice (transferability) rather than internal validity (credibility)

www.joannabriggs.edu.au

Considerations in Data Extraction

• Source - citation and contact details• Eligibility - confirm eligibility for review• Methods - study design, concerns about bias• Participants - total number, setting, diagnostic criteria • Interventions - total number of intervention groups• Outcomes - outcomes and time points• Results - for each outcome of interest: sample size, etc• Miscellaneous - funding source, etc

www.joannabriggs.edu.au

Data Extraction• The data extracted for a systematic review are the

results from individual studies specifically related to the review question.

• Difficulties related to the extraction of data include:– different populations used– different outcome measures– different scales or measures used– interventions administered differently– reliability of data extraction (i.e: between reviewers)

www.joannabriggs.edu.au

Extracting Findings for Qualitative reviews• The units of extraction in this process are specific

findings and illustrations from the text that demonstrate the origins of the findings

• A finding is defined as: A conclusion reached by the researcher(s) and often presented as themes or metaphors

www.joannabriggs.edu.au

Analyses• Depend on data collected/question• Maybe simply narrative• Meta analysis?• Qualitative syntheses?• Mixed methods

www.joannabriggs.edu.au

Analyses and reporting– What interventions/activities have been evaluated– The effectiveness /appropriateness /feasibility

/meaningfulness of the intervention/activity– Contradictory findings and conflicts– Limitations of study methods– Issues related to study quality– The use of inappropriate definitions– Specific populations excluded from studies– Future research needs

Meta Analysis &Meta Synthesis

1004 references

832 referencesScanned Ti/Ab

172 duplicates

117 studiesretrieved

715 do not meetIncl. criteria

82 do not meetIncl. criteria

35 studies forCritical Appraisal

26 studies incl.in review

6 studies incl.in meta analysis

20 studies incl.in narrative

9 excludedstudies

www.joannabriggs.edu.au

Statistical methods for meta-analysis• Quantitative method of combining results of

independent studies• Aim is to increase precision of overall estimate• Investigate reasons for differences in risk estimates

between studies• Discover patterns of risk amongst studies

www.joannabriggs.edu.au

When is meta-analysis useful?• If studies report different treatment effects.• If studies are too small (insufficient power) to detect

meaningful effects.• Single studies rarely, if ever, provide definitive

conclusions regarding the effectiveness of an intervention.

www.joannabriggs.edu.au

When meta-analysis can be used• Meta analysis can be used if studies:

– have the same population– use the same intervention administered in the same way.– measure the same outcomes

• Homogeneity– studies are sufficiently similar to estimate an average

effect.

Taken from Egger, M. et al. BMJ 1998;316:140-144

Example meta-analysis

www.joannabriggs.edu.au

• Analysis and synthesis of qualitative studies • Based on processed data• Aim of meta-synthesis

• is to assemble findings; • categorise these findings into groups on the basis of

similarity in meaning; • aggregate these to generate a set of statements that

adequately represent that aggregation.

Meta-Synthesis

www.joannabriggs.edu.au

Data Synthesis Involves:• Step 1: Identifying findings• Step 2: Grouping findings into categories; and• Step 3: Grouping categories into synthesised findings

STEP 2: CATEGORIES

STEP 3: SYNTHESISED FINDINGS

FIRST ORDER ANALYSIS

SECOND ORDER INTERPRETATION

THIRD ORDER INTERPRETATION

QARI METAGGREGATION

META ETHNOGRAPHY

Comprehensive/Mixed method Review

FOCUS technical brief #25, Angela Harden, 2010

www.joannabriggs.edu.au

Interpretation• Quality, strength and applicability of evidence for

outcomes• Applicability for stakeholders/policy makers• Qualitative evidence can aid in interpretation• How, why, barriers, facilitators, experience• Conclusions/recommendations should be drawn

based on available evidence• May be none available!

The JBI Software

System for theUnified

Management of theAssessment and

Review ofInformation

Consists of the following components

Comprehensive Review and Management System

Keeps all review information together and generates a

report that may contain elements from all other

SUMARI modules

• Protocol• Reviewers• Search strategy• Bibliography - retrieved studies• Bibliography - non selected

studies

Qualitative Assessment and Review Instrument

Meta-Analysis of Statistics Assessment and Review Instrument(Used in place of Review Manager for non-Cochrane reviews)

Narrative Opinion and Text Assessment and Review Instrument

Analysis of Cost Technology and Utilisation Assessment and Review Instrument