Quality and Reporting Characteristics
of Network Meta-analyses: A Scoping Review
Andrea C. Tricco MSc, PhD
Scientist, Li Ka Shing Knowledge Institute of St. Michael’s Hospital
Assistant Professor, Dalla Lana School of Public Health, University of Toronto
CADTH Symposium 2016
Conflict of interest statement:
I have no actual or potential conflict of interest in relation to this presentation.
Funding:
This research was funded by the Canadian Agency for Drugs and Technologies in Health.
Definition of Network Meta-analysis (NMA)
NMA is an extension of indirect comparisons that allows the combination of direct with indirect comparisons, and also the simultaneous analysis of the comparative effects of many interventions
3
Cochrane Handbook, http://handbook.cochrane.org/
Network meta-analysis (NMA)
4
Steps for a systematic review in a NMA
5
Start with a systematic review and NMA protocol
Conduct a comprehensive literature search
Use pre-defined eligibility criteria, same types of data abstracted
Use the same risk of bias appraisal as recommended for reviews
Synthesize totality of evidence and report using PRISMA-NMA
Discuss strengths and limitations
Conduct each step by 2 reviewers, independently
Hutton et al., Ann Int Med, 2015
Methodological considerations
6
Broader PICOS than systematic reviews with a meta-analysis
Literature search must include all interventions/comparators
= Expect to include many more studies!
Decisions on how to “lump” and “split” treatments into nodes (e.g., dosing, administration, drug class versus specific drug analysis)
= A lot of clinical guidance!
Statistical considerations
7
Network geometry
Key assumptions:
1. Homogeneity – similar considerations as meta-
analysis
2. Similarity (transitivity) – effect modifiers
3. Consistency – agreement between direct and indirect
evidence
Assumptions underlying indirect
comparisons and NMA
Cipriani et al Ann of Int Medicine 2013
Assumption for indirect and mixed comparison
Conceptual definition (Transitivity)
Clinical Methodological
Property of parameters and data
(Consistency)
Statistical
Transitivity
Interpretation of transitivity:
1. Treatment A is similar when it appears in AB and AC
trials
2. The two sets of trials AB and AC do not differ with
respect to the distribution of effect modifiers
3. Participants included in the network could in principle
be randomized to any of the 3 treatments A, B, C
4. ‘Missing’ treatment in each trial is missing at random
5. There are no differences between observed and
unobserved relative effects of AB and AC beyond
what can be explained by heterogeneity
Salanti Res Synth Methods 2012
A
B
C
Consistency
B
C
A
B
C
Direct and indirect evidence are in
agreement
10
Consistency is a property of a
closed loop (path that starts and
ends at the same node)
Heterogeneity
The heterogeneity variance can be estimated using several approaches
─ E.g., DerSimonian and Laird (DL), Maximum Likelihood (ML), Restricted
Maximum Likelihood (REML)
In a Bayesian framework several priors can be assigned
─ E.g., Informative, Minimally informative, Vague
Several heterogeneity assumptions can be considered
1. Common within-network heterogeneity: all treatment comparisons in the
network are associated with the same magnitude of heterogeneity
2. Comparison-specific heterogeneity: each treatment comparison in the
network has its own amount of heterogeneity
3. Common within-loop heterogeneity: treatment comparisons included in each
closed loop in the network share the same amount of heterogeneity
11
Objective
To explore the quality and reporting characteristics
of all published NMAs through a scoping review
12
Methods
A priori protocol compiled using PRISMA-P
Methods guided by Joanna Briggs Institute Methods Manual
MEDLINE, EMBASE, PubMed and Cochrane searched from
inception to April 14, 2015 without any language or
publication status restrictions
Included NMAs with 4 treatments, randomized trials, and
using a valid statistical analysis
Pairs of reviewers independently performed screening of
citations and full-text articles; abstracted data; and assessed
quality of included studies (AMSTAR and ISPOR)
13
Results – study flow
Additional records identified through other sources
(n = 253)
Scre
en
ing
Incl
ud
ed
Elig
ibili
ty
Ide
nti
fica
tio
n
Citations identified through database
searching
(n = 3727)
Citations after duplicates removed (n = 3538)
Citations screened (n = 3538)
Records excluded (n = 2913)
•Not an NMA (n=2601) •NMA includes observational studies (n=61) •Includes <4 treatments (n=216) •Commentary/Protocol (n=35)
Full-text articles assessed for eligibility
(n = 625)
Full-text excluded (n = 382)
•Unable to locate (n = 3) •Duplicate (n=16) •(Commentary/Protocol (n = 91) •Invalid NMA (n = 98) •NMA includes observational studies (n = 21) •Includes <4 treatments (n = 70) •No. of treatments > no. of studies (n = 76) •Companion report (n = 7)
Studies included in qualitative synthesis (n = 496)
Full-text articles assessed for eligibility
(n = 878)
Results – geographic trends
Geographic distribution of publications by region (n=496)
31%
0.6%
1.2%
1.4%
North America
Africa
Central &
South America
Australia
13% Europe
52%
Asia
Multiple regions: 0.4%
0.2% 0.2% 0.4% 0.2% 0.2% 0.4% 1.0%
2.8%
6.3% 6.7%
11.7% 12.5%
18.3%
23.8%
10.5%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
1997 1999 2000 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015*
Results – temporal distribution
Temporal distribution of publications (n=496)
2.4% 2.4%
*2015 sample is not complete
Results – frequency of terms
Word Cloud of NMA terminology based on frequency (n=496)
NMA terminology (n=496) Count (%)
Network meta-analysis 232 (38.9%)
Mixed treatment comparisons 122 (20.5%)
Indirect comparisons 57 (9.6%)
Results – cited methodology
Word cloud of cited methodology paper (n=496)
Most Commonly Cited Papers Count (%)
Lu & Ades 2004 149 (30%)
Caldwell 2005 82 (17%)
Bucher 1997 71 (14%)
Results – journal disciplines
Journal Discipline (n= 496) Count (%)
Medicine, General & Internal 121 (23.8%)
Health Care Sciences & Services 56 (11.0%)
Pharmacology & Pharmacy 35 (6.9%)
Cardiac & Cardiovascular Systems 29 (5.7%)
Endocrinology & Metabolism 25 (4.9%)
Oncology 21 (4.1%)
Multidisciplinary Sciences 19 (3.7%)
Rheumatology 17 (3.3%)
Clinical Neurology 15 (2.9%)
Gastroenterology & Hepatology 15 (2.9%)
Results – NMA characteristics
NMA Characteristics (n=496) Count (%)
Funding
Publicly-sponsored 186 (38%)
Industry-sponsored 103 (21%)
Non-sponsored 106 (21%)
Industry and publicly sponsored 9 (2%)
Funding source not reported 92 (19%)
Article Type
Application paper 438 (88%)
Application paper (no review method) 18 (4%)
Development paper 40 (8%)
Review Size (n=438)* median (Q1, Q3) 25 (14, 48)
*Review size reported for application papers with full review methods only
Results – review design
Method Characteristics (n=438) Count (%)
Re
vie
w T
yp
e
Knowledge Synthesis
Name
Systematic review 356 (81.3%)
Overview of reviews 7 (1.6%)
Not reported 75 (17.1%)
Review Duration
<6 months 55 (12.6%)
6-12 months 132 (30.1%)
>12-24 months 106 (24.2%)
>24 months 81 (18.5%)
Not reported 64 (14.6%)
A p
rio
ri p
roto
co
l a
nd
revie
w d
es
ign
A Priori Protocol
Use of a protocol mentioned 66 (15.1%)
Published 40 (9.1%)
Registered 25 (5.7%)
Available upon request 6 (1.4%)
Not reported 301 (68.7%)
Research Question Clearly Reported 437 (99.8%)
Unclear/inferred 1 (0.2%)
Eligibility Criteria
Clearly reported 430 (98.2%)
Unclear/inferred 1 (0.2%)
Not reported 7 (1.6%)
Results – literature search
Method Characteristics (n=438) Count (%)
Iden
tify
ing
rele
van
t stu
die
s
Search String
Complete literature search 207 (47.3%)
Medical subject headings only 173 (39.5%)
Not reported 58 (13.2%)
Databases Searched
Searched >1 database 407 (92.9%)
Searched only 1 database 29 (6.6%)
Not reported 2 (0.5%)
Additional Search
Strategy
Scanned references 309 (70.5%)
Grey literature searched 270 (61.6%)
Consulted topic experts 80 (18.3%)
Consulted librarian 67 (15.3%)
Performed updated search 62 (14.2%)
Manually searched select journals 37 (8.4%)
Limits Applied
Limited by date 135 (30.8%)
Limited by language 147 (33.6%)
Limited by study design 291 (66.4%)
Other limits (e.g., age, humans) 129 (29.5%)
Results – screening, abstraction, quality
Method Characteristics (n=438) Count (%)
Stu
dy S
ele
cti
on
Title & Abstract Screening
Reported 402 (91.8%)
Not reported 36 (8.2%)
Full-text Screening
Reported 405 (92.5%)
Not reported 33 (7.5%)
Study flow
Completely in PRISMA-like flow diagram 374 (85.4%)
Completely in text/table only 20 (4.6%)
Partially reported 15 (3.4%)
Not reported 29 (6.6%)
Da
ta
Ab
str
ac
tio
n &
Qu
ali
ty
As
se
ss
me
nt Data Abstraction
Reported 414 (94.5%)
Not reported 24 (5.5%)
Quality Appraisal
Reported 346 (79.0%)
Not reported 92 (21.0%)
65%
2%
4%
0.5%
21%
8%
64%
3%
2%
0.2%
24%
7%
54%
21%
2%
0.0%
17%
5%
41%
5%
2%
0.5%
30%
21%
0% 10% 20% 30% 40% 50% 60% 70%
≥ 2 independent reviewers
1 reviewer & 1 verifier
1 reviewer only
Not done
Done but unclear # of reviewers
Not reported
≥ 2 independent reviewers
1 reviewer & 1 verifier
1 reviewer only
Not done
Done but unclear # of reviewers
Not reported
≥ 2 independent reviewers
1 reviewer & 1 verifier
1 reviewer only
Not done
Done but unclear # of reviewers
Not reported
≥ 2 independent reviewers
1 reviewer & 1 verifier
1 reviewer only
Not done
Done but unclear # of reviewers
Not reported
Tit
le &
Ab
str
act
Scre
en
ing
Fu
ll-t
ext
Scre
en
ing
D
ata
Ab
str
acti
on
Qu
ality
Assessm
en
t Results – conduct of review process
Method characteristics – conduct of review process (n=438)
Short-cuts taken (6%)
Short-cuts taken (4%)
Short-cuts taken (23%)
Short-cuts taken (7%)
Inadequate reporting (29%)
Inadequate reporting (31%)
Inadequate reporting (22%)
Inadequate reporting (51%)
Results – streamlined methods
NMAs that leveraged previous systematic review(s) Count (%)
Leveraged previous systematic reviews (n=456)*
Yes 78 (17%)
No 378 (83%)
Leverage Process (n= 78)
Updated literature search of previous systematic review(s) 37 (47%)
Used literature database of previous systematic review(s) 20 (26%)
Updated and expanded literature search of previous systematic review(s) 9 (12%)
Cochrane update 6 (8%)
Used database of previous systematic review(s) 3 (4%)
Expanded literature search of previous systematic review(s) 2 (3%)
Updated literature search of previous systematic review(s) and used data from previous
review 1 (1%)
*Includes all application papers
Results – AMSTAR components
Review methodological quality – AMSTAR (n=438)
31%
58%
84%
62%
18%
83%
62%
48%
86%
36%
16%
69%
39%
0.5%
33%
1%
13%
21%
1%
12%
0.5%
3%
15%
5%
82%
5%
17%
29%
2%
63%
84%
21%
0% 20% 40% 60% 80% 100%
A priori design
Duplicate screening and data extraction
Comprehensive literature search
Grey Literature
List of includes / excludes
Study characteristics
Quality Appraisal
Quality of evidence incorporated in conclusions
Appropriate pooling methods
Publication bias assessed
Conflict of interest declared
YES UNCLEAR NO NOT APPLICABLE
Results – AMSTAR scores overall
Review methodological quality – AMSTAR overall quality (n=438)
1% 1%
7%
8%
13% 12%
17%
15%
11%
7%
4% 3%
0%
4%
8%
12%
16%
20%
0 1 2 3 4 5 6 7 8 9 10 11
Low Quality Moderate Quality High Quality
Sc
ore
Dis
trib
uti
on
(%
)
AMSTAR Score
median (IQR)= 6 (4,7)
Low: 18%
Moderate: 57%
High: 25%
0%
20%
40%
60%
80%
100%
1999 2000 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Year of Publication
Low (0-3) Moderate (4-7) High (8-11)
100%
50%
100%
50%
21%
13%
31% 29% 25% 22% 19%
10% 4%
50%
100%
25%
64%
50%
90%
50%
64%
62% 59%
71%
79%
74%
25%
14%
38%
10%
19%
7% 13%
19%
10% 11%
21%
Results – AMSTAR scores over time
Review methodological quality vs. publication year (n=438)
Results – ISPOR components
NMA methodological quality – ISPOR (n=456)
54%
48%
31%
81%
12%
46%
21%
41%
73%
20%
20%
1%
19%
24%
27%
31%
27%
14%
10%
0% 20% 40% 60% 80% 100%
Consistency Assessed
Direct & indirect evidence combined
Accounted for inconsistency
Valid rationale for FE or RE model
Heterogeneity assumption explored
Accounted for heterogeneity
YES UNCLEAR NO Not Applicable
Summary
Sub-optimal quality of reporting was observed
PRISMA-NMA may lead to improvements in reporting over time
Less than a quarter of the published NMAs were considered high
quality (AMSTAR ≥8), however, this improved over time
Authors of published NMAs used streamlined methods:
23% streamlined data abstraction, 17% leveraged previous reviews
About 20% did not assess consistency, provide rationale for FE/RE model or
account for heterogeneity
ISPOR may lead to improvements in review/NMA conduct
Acknowledgements
Co-investigators on funding proposal: Dr. Sharon Straus, Dr. Areti Angeliki Veroniki
Research team:
Wasifa Zarin
Vera Nincic
Afshin Vafaei
Dr. Shannon Sullivan
Dr. Emily Reynen
Sanober Motiwala
Maria Petropoulou
CIHR (new investigator award, MAGIC team grant)
31
Myrsini Gianiasti
Jesmin Antony
Caitlin Daly
Joycelyne Ewusie
Dr. Adriani Nikolakopoulou
Dr. Anna Chaimani
Dr. Georgia Salanti
Questions?
Email: [email protected]
Office Location:
209 Victoria Street, 7th floor, Room 721, East building, Toronto, ON
Telephone: 416.864.6060 x77521
32
Top Related