LIBRARIAN INVOLVEMENT IN SYSTEMATIC REVIEWS Mark MacEachern & Whitney Townsend.
SYSTEMATIC REVIEWS: A PRIMER FOR LIBRARIANS Mark MacEachern, MLIS Taubman Health Sciences Library,...
-
Upload
shannon-lindsey -
Category
Documents
-
view
219 -
download
0
Transcript of SYSTEMATIC REVIEWS: A PRIMER FOR LIBRARIANS Mark MacEachern, MLIS Taubman Health Sciences Library,...
SYSTEMATIC REVIEWS: A PRIMER FOR LIBRARIANS
Mark MacEachern, MLIS
Taubman Health Sciences Library, University of Michigan
Outline
Introductions What is a Systematic Review? Librarian Roles Protocols & Registries Databases & Search Strategies Data Management Reporting Wrap Up & Other Stuff
Introduction
Name Experience with systematic reviews What do you want to get from this?
Ask questions throughout! I have no conflicts of interest to report Some overlap with my MLA webinars and the
UMich workshop Discussion-based
What is a systematic review?
From the Cochrane Handbook (1.2.2):
A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993).
What is a systematic review?
Common Elements: A clearly stated set of objectives with pre-defined
eligibility criteria for studies A reproducible methodology A systematic search that attempts to identify all
studies that would meet the eligibility criteria An assessment of the validity of the findings of the
included studies, for example through the assessment of risk of bias
A systematic presentation, and synthesis, of the characteristics and findings of the included studies
Source: Cochrane Handbook, 1.2.2
What is a systematic review?
Source: Gagnier JJ. Introduction to Systematic Reviews & Meta-analyses.
In short, it is likely a narrative review if: The question is unclear from the title, abstract, or
introduction There is no methods section
What is a systematic review?
In short, it is likely a narrative review if: The question is unclear from the title, abstract, or
introduction There is no methods section
Types of Systematic Reviews Qualitative
Results summarized narratively, no stats, methods rigorous to minimize bias
Quantitative Results statistically combined, methods rigorous to
minimize bias
Source: Gagnier JJ. Introduction to Systematic Reviews & Meta-analyses.
What is a systematic review?
Steps1. Formulate a question2. Define inclusion/exclusion criteria3. Locate studies4. Select studies5. Assess study quality6. Extract data7. Analyze and present results8. Interpret results
Source: Gagnier JJ. Introduction to Systematic Reviews & Meta-analyses.
What is a systematic review?
Source: Sackett DL. Evidence-based medicine: how to practice and teach EBM.
What is a systematic review?
Team Composition One person
Possible, but more prone to bias Two people
May want a third to help with disagreements Three plus
Clinician(s), statistician, librarian Too many can become problematic
Source: Gagnier JJ. Introduction to Systematic Reviews & Meta-analyses.
What is a systematic review?
Source: Institute of Medicine. Standards for Initiating a Systematic Review.
Team Composition
What is a systematic review?
Source: Khangura S. Evidence summaries: the evolution of a rapid review approach.
Rapid Reviews
What is a systematic review?
Source: EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews
Methodological Approaches
Search fewer databases
Limited use of grey literature
Restrict types of studies included (eg., English only, most recent 5 yrs)
Relying on existing reviews
Limited full-text review
Limited dual review for study selection
See report for more
Rapid Reviews
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.
A researcher comes to you set on doing a systematic review for a specific question. You don’t believe that the question lends itself to a systematic review. How do you proceed?
RAPID FIRE SCENARIOSIn groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.
A resident reaches out to you for help conducting a systematic review search. She tells you that the project is done, the manuscript is finished, and all she needs is a search that captures every study included in her paper. Is this a problem? How do you proceed here?
Librarian Roles
IOM Standards for Initiating a Systematic Review Standard 2.1 Establish a team with appropriate expertise
and experience to conduct the systematic review 2.1.3 Include expertise in searching for relevant
evidence Standard 2.5 Formulate the topic for the systematic
review 2.5.1 Confirm the need for a new review
Standard 2.6 Develop a systematic review protocol 2.6.4 Describe the search strategy for identifying
relevant evidence
Source: Institute of Medicine. Standards for Initiating a Systematic Review
Librarian Roles
IOM Standards for Finding and Assessing Individual Studies Standard 3.1 Conduct a comprehensive systematic
search for evidence 3.1.1 Work with a librarian or other information specialist
trained in performing systematic reviews to plan the search strategy
3.1.2 Design the search strategy to address each key research question
3.1.3 Use an independent librarian or other information specialist to peer review the search strategy
3.1.4 – 3.1.9 Search various databases and update searches
Source: Institute of Medicine. Standards for Initiating a Systematic Review.
Librarian Roles
IOM Standards for Finding and Assessing Individual Studies Standard 3.2 Take action to address potentially biased
reporting of research results 3.2.1 Search grey literature databases, clinical trial
registries, and other sources of unpublished information about studies
3.2.4 – 3.2.6 Handsearch, web search, and non-English search
Standard 3.4 Document the search 3.4.1 Provide a line-by-line description of the search strategy
Source: Institute of Medicine. Standards for Initiating a Systematic Review.
Librarian Roles
Common Errors in Published SR Search Strategies
Spelling errors Misuse of MeSH and keywords
No spelling variants / truncations
Bad exploding
Incorrect use of Boolean and line numbers
Redundancy in search terms
Translating strategies poorlySource: Sampson, M. Errors in search strategies were identified by type
and frequency.
Source: Institute of Medicine. Standards for Systematic Reviews (full report)
Librarian Roles
Source: ICMJE. Defining the Role of Authors and Contributors
Authorship – ICMJE Authorship Criteria1. Why Authorship Matters?
Credit & responsibility for contributions
2. Who Is an Author? Substantial contributions to all aspects of article Drafting and revising manuscript Final approval of manuscript Agreement to be accountable for all aspects of the
work
3. Non Author Contributions If you don’t meet those 4 criteria, you should be listed
in the acknowledgements
Librarian Roles
Source: ICMJE. Defining the Role of Authors and Contributors
Authorship – ICMJE Authorship Criteria1. Why Authorship Matters?
Credit & responsibility for contributions
2. Who Is an Author? Substantial contributions to all aspects of article Drafting and revising manuscript Final approval of manuscript Agreement to be accountable for all aspects of the
work
3. Non Author Contributions If you don’t meet those 4 criteria, you should be listed
in the acknowledgements
Librarian Roles
Source: ICMJE. Defining the Role of Authors and Contributors
Authorship – ICMJE Authorship Criteria1. Why Authorship Matters?
Credit & responsibility for contributions
2. Who Is an Author? Substantial contributions to all aspects of article Drafting and revising manuscript Final approval of manuscript Agreement to be accountable for all aspects of the
work
3. Non Author Contributions If you don’t meet those 4 criteria, you should be listed
in the acknowledgements
Librarian Roles
Grants Co-PI, Collaborator, Consultant % effort, compensation Good source for demonstrating value
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.
Is authorship on SRs appropriate? If so, in what instances? If not, why not?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.
You have been asked to participate in a systematic review project and would like to receive appropriate recognition for your contributions. How would you broach this subject with the project lead?
Protocols & Registries
A protocol is a plan or set of steps to be followed in a study. A protocol for a systematic review should describe the rationale for the review; the objectives; and the methods that will be used to locate, select and critically appraise studies, and to collect and analyse data from the included studies.
Source: Cochrane Community, http://community.cochrane.org/faq/general#t86n544 Courtesy: W Townsend
Protocols & Registries
Source: Liberati A. The PRISMA Statement…Explanation and Elaboration.
Prevents post-hoc changes, but acknowledges that changes are permissible. Don’t feel like it sets everything in stone.
Prevents duplication of effort
Protocols & Registries
Educate • Ensure consults know what goes into an SR. Filters out those who do not.
• Proper SR methodologies• Realistic timelines, realistic questions• Scoping review first• Technology• Reference IOM, Cochrane, PRISMA
Protocols & Registries
Educate • Ensure people know what goes into an SR. Filters out those who don’t.
• Proper SR methodologies• Realistic timelines, realistic questions• Scoping review first• Technology• Reference IOM, Cochrane, PRISMA
Inform • Your search• Helps elicit important info from project lead
• PICO• Eligibility criteria• Possible limits• Appropriate databases
Protocols & Registries
Educate • Ensure people know what goes into an SR. Filters out those who don’t.
• Proper SR methodologies• Realistic timelines, realistic questions• Scoping review first• Technology• Reference IOM, Cochrane, PRISMA
Inform • Your search• Helps elicit important info from project lead
• PICO• Eligibility criteria• Possible limits• Appropriate databases
Distribute • Makes the project team accountable• Facilitates team functioning• Helps with the writing
Protocols & Registries
Source: Rethlefsen ML. Librarian co-authors correlated with higher quality reported...
Protocols & Registries
Realistic timelines
Source: Cochrane Handbook, 2.3.b
In 2008: 15 months (Sampson M, 2008)
In 2013: 8 months (Beller EM, 2013 )
Protocols & Registries
Why register? Provides transparency Reduce risk of multiple reviews on the same
question Reduce publication bias Safeguard against reporting biases Identify similar reviews in the process of being
completed
Source: Liberati A. The PRISMA Statement…Explanation and Elaboration.
Protocols & Registries
Source: Sun G. Systematic review of quality improvement initiatives in the intensive care setting
Protocols & Registries
Source: Sun G. Systematic review of quality improvement initiatives in the intensive care setting
Protocols & Registries
Source: Sun G. Systematic review of quality improvement initiatives in the intensive care setting
Protocols & Registries
Review title and timescale
Review title*
Original language title
Anticipated or actual start date*
Anticipated completion date*
Stage of review at time of registration*
Review team details
Named contact*
Named contact email*
Named contact address
Named contact phone number
Review team members and their organisational affiliations
Organisational affiliation of the review*
Funding sources/ sponsors*
Conflicts of interest*
Collaborators
Review methods
Review question(s)*
Searches*
URL to search strategy
Condition or domain being studied*
Participants/ population*
Intervention(s), exposure(s)*
Comparator(s)/ control*
Types of study to be included initially*
Context
Primary outcome(s)*
Secondary outcomes*
Data extraction (selection and coding)
Risk of bias (quality) assessment*
Strategy for data synthesis*
Analysis of subgroups or
subsets*
General information
Type of review
Language
Country
Other registration details
Reference and/or URL for published protocol
Dissemination plans
Keywords
Details of any existing review of the same topic by the same authors
Review status*
Any other information
Link to publication of final report
Source: PROSPERO, http://www.crd.york.ac.uk/PROSPERO/
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.
You and project lead have a discussion about protocols and are trying to decide on whether to have one or not. She says that most don’t have one, so why should we bother? This gets you thinking about whether they’re worth the effort? Do you care whether the project has one or not?
Databases & Search Strategies
Why is the search important? A high-quality lit search is essential for a successful
meta-analysis It is from the search results that data is gathered for
analysis Failure to locate important studies can significantly
affect results Remember the goal is to capture every relevant
study Important to report search so your methodology
can be reproduced
Databases & Search Strategies Expect:
1000s of results To search multiple databases
Ovid MEDLINE, EMBASE, PubMed, Scopus, etc. The search process to take awhile To publish search strategy and search methodology
Databases & Search Strategies
Core Secondary Others
• MEDLINE• PubMed• Ovid
MEDLINE• EMBASE• CENTRAL
• Scopus• Web of Science• CINAHL• Google Scholar• ClinicalTrials.go
v
• PsycINFO• ERIC• Sociological
Abstracts• ABI Inform• Proquest
Theses and Dissertations
• Etc.
• Cochrane Handbook (10.3.1)– Search multiple sources– Search unpublished studies– Search trial registries
Databases & Search Strategies Supplemental strategies
Examine the references of included studies and relevant reviews
Use ‘cited by’ features Web of Science, Google Scholar
Contact authors, companies, orgs, societies, etc. Hand search important journals
By consensus, Impact Factor, most frequently appearing journals in Endnote/etc
Search for ongoing studies Clinicaltrials.gov ; controlled-trials.com (ISRCTN);
conference abstracts
Databases & Search Strategies Grey Literature
“It is usually understood to mean literature that is not formally published in sources such as books or journal articles.” (Cochrane Handbook, 6.2.1.8)
Types Conference abstracts (Web of Science, Scopus) Clinical trials (ClinicalTrials.gov) Government reports, documents (.gov, CABI) Dissertations (ProQuest Dissertations and Theses) Unpublished manuscripts
Databases & Search Strategies
Pros Cons
• Combats publication bias• Process can help find
missed studies• Strengthens methodology
• Search functionality of grey lit sources tends to be basic
• Difficult sell value to team• Increases retrieval (already
large)• Adds time to project
(contact authors, etc.)• Difficult to export results
Grey Literature
Databases & Search Strategies
Pros Cons
• Combats publication bias• Process can help find
missed studies• Strengthens methodology
• Search functionality of grey lit sources tends to be basic
• Difficult sell value to team• Increases retrieval (already
large)• Adds time to project
(contact authors, etc.)• Difficult to export results
Grey Literature
Cook DJ, 1993 – Unpublished studies should not be systematically excluded
Hopewell S, 2002 – Useful for RCTs. Combine with MEDLINE
Egger M, 2003 – Difference b/w comprehensive searches and less comprehensive is small
Martin JL, 2005 – Study quality is important and grey lit might not meet the standard
Armstrong R, 2005 – Handsearching is valuable to pick up supplements
Benzies KM, 2006 – Grey literature is useful
Hopewell S, 2007 – Handsearching minimizes bias, but is costly and requires much effort
Hopewell S, 2009 – Publication bias is a thing; consider methods to minimize it
Craane B, 2012 – Handsearching contributes considerably
Mahmood Q, 2014 – Searching for grey lit has benefits and challenges
Saleh AA, 2014 – Average time is 7 hours of librarian time to search grey lit sources
Cochrane Handbook, 10.3.2 – Publication bias is a major threat. Including unpublished trials appears to help
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR teams.
The project lead does not want to search the grey literature. Is grey literature worth the effort? Is it something you would insist on doing?
Databases & Search Strategies Harvesting Search Terms
PICO Protocols Sentinel and other relevant articles
Controlled terms Read titles and abstracts Find others via (***.ti AND ***.ti) Entry terms
Use adjacency to cover variations E.g. (breast* adj5 (cancer* or carcinom* or neoplas*
or tumor* or tumour* or malignan*)).tw
Filters A predefined search designed to target specific
study methodologies (RCTs, Cohort, Systematic Reviews)
Consider using a validated filter E.g.: PubMed Clinical Queries – Specific, Therapy
Databases & Search Strategies
Databases & Search Strategies
Eligibility criteria can inform filters Be careful applying them
RCT publication type vs Title searches
Databases & Search Strategies
Eligibility criteria can inform filters Be careful applying them
Humans limit?
Databases & Search Strategies
Sources of validated filters PubMed Clinical Queries
http://www.ncbi.nlm.nih.gov/books/NBK3827/#pubmedhelp.Clinical_Queries_Filters
PubMed Special Queries http://www.nlm.nih.gov/bsd/special_queries.html
HIRU (McMaster University) http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx
SIGN http://www.sign.ac.uk/methodology/filters.html
InterTASC http://www.york.ac.uk/inst/crd/intertasc/
Cochrane Groups Individual studies
http://www.ncbi.nlm.nih.gov/pubmed/19712211 (eg)
Databases & Search Strategies
Supplemental strategies Handsearching journals (see Cochrane 6.2.2.1)
Those selected by team Impact Factor Most frequently appearing journals in Endnote/Excel
Databases & Search Strategies Validate searches
Important to validate your search in each database Make sure search captures all sentinel articles
If search doesn’t, why? How:
Databases & Search Strategies Cochrane recommendations
Source: Cochrane Editorial Unit, Methodological Expectations of Cochrane Intervention Reviews
Databases & Search Strategies Cochrane recommendations
Source: Cochrane Editorial Unit, Methodological Expectations of Cochrane Intervention Reviews
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR teams.
The team tells you directly that they are only interested in searching one database. What are the problems with this approach and how would you discuss these problems with the project lead/team?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR teams.
I’m getting way too many results and I’m thinking of using NOT to exclude editorials, letters, adults, and animals. What are possible limitations to applying so many filters? Are there other approaches? Other considerations?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenario and identify solutions to this problem that commonly arises in SR work.
Let’s say you limited your search to RCTs, English language studies, humans, and to the date range 2001-2015. How do you apply such restrictions to databases with few filtering options (e.g., Scopus, Web of Science, PsycINFO, ClinicalTrials.gov, etc.)?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenario and identify solutions to this problem that commonly arises in SR work.
I’ve been working on a search that aims to identify trends within the cancer literature. It’s not a systematic review, but requires a significant search. I try applying PubMed’s Cancer Subset search, but find it to be much too inclusive. Is it okay to create my own? Is it okay to use an unvalidated filter?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR teams.
As you are building out your keyword searches, you are considering whether to rely on title or title/abstract searches. Or, .mp in Ovid MEDLINE? What do you factor in when making a decision?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR teams.
The project lead says that he doesn’t want to sift through more than 200 papers. What are the issues and how would you handle this?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR teams.
You’ve completed the searches and suggested to the team that they check the reference lists and do a Times Cited check of their included studies. As the team does this they notice that one of the papers has been cited over 500 times. They don’t want to consider these new papers. How do you handle this situation?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR teams.
The project lead is most familiar with PubMed and would like you to run the search in that database. You feel Ovid MEDLINE is more appropriate. What are some of the differences between the two resources? How would you discuss these issues with the project lead?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR teams.
You’ve been focused on a specific systematic review search for a long time and worry that there are minor mistakes. Would you consider recruiting a peer-reviewer? What if you can’t find one? If you were the peer-reviewer, how much time would you anticipate to spend reviewing someone else’s search?
Data Management
An SR is an extensive search with greater emphasis on data mgmt
Programs Endnote, Excel, DistillerSR, Abstrakr, other tools
Uses Keeping track of citations from multiple databases Eliminating duplicates Sharing citations with team Storing pdfs Eligibility screening Exporting to word, excel, and other formats Creating in-text citations and bibliographies
Data Management
• What to note:• Databases searched• Dates searches were
run• All searches
• Result numbers (total & unique)
• Sentinel articles used for testing
• Decide on naming convention
Data Management
Source: Cochrane Handbook, 7.2.3
Data management strategy informs screening process
Cochrane recommends title/abstract review, then full-text
Data Management
Source: Institute of Medicine, Standards for Initiating a Systematic Review
Additional screening recommendations
RAPID FIRE SCENARIOS
In groups of 3, consider the scenarios and identify solutions to the problems/issues that commonly arise in SR work.
The project lead and/or the team does not want to learn Endnote for this project. What are possible solutions?
RAPID FIRE SCENARIOS
In groups of 3, consider the scenarios and identify solutions to these problems/issues that commonly arise in SR work.
The screening team has identified 84 papers for full-text review and has asked for help obtaining the pdfs. How would you handle this?
RAPID FIRE SCENARIOS
In groups of 3, consider the scenarios and identify solutions to these problems/issues that commonly arise in SR work.
You’re working on an SR that includes members from multiple institutions. How do you manage the citations with a team comprised of folks who may or may not have the same access to resources?
Reporting
Source: PRISMA, http://www.prisma-statement.org/statement.htm
PRISMA: aims to ensure the transparent and complete reporting
Reporting
Excerpts from the Explanation:
• Retrieval from databases may be imperfect, even if done by experienced searchers
• List database, platform, or provider
• Include start and end dates• Authors should report who
developed and conducted the search
• Report supplementary approaches, incl hand searching, reference lists, trial registries, contacting authors
Source: Liberati, 2009
PRISMA: Item 7 – Information sources
Reporting
Excerpts from the Explanation:
• The search strategy is an essential component of any SR
• Note if the search is peer-reviewed
• It’s not always feasible to include the full strategy in the paper. Include it as an appendix, online supplement, etc.
• Archive searches• Note limits and other
limitations• Database inaccessibility• Financial / time
restrictions• No searching expertise
Source: Liberati, 2009
PRISMA: Item 8 – Search
Excerpts from the Explanation:
• Report total number of records identified from all sources
• Include counts for each source, b/c literature identified primarily from references and experts can be biased
• Note: • Duplicates• Unique records• Levels of screening
• Layout of diagram may vary
Source: Liberati, 2009
PRISMA: Flow Diagram
Reporting
Reporting
See Cochrane Handbook http://www.cochrane-handbook.org/
Section 6.6
“It should be borne in mind at the outset that the full search strategies for each database will need to be included in an Appendix of the review.”
Reporting
See Cochrane Handbook http://www.cochrane-handbook.org/
Section 6.6
In study flow diagram: number of unique records identified by the searches; number of records excluded after preliminary screening (e.g. of
titles and abstracts); and number of records retrieved in full text
Reporting
See Cochrane Handbook http://www.cochrane-handbook.org/
Section 6.6
In abstract: List all databases searched. Note the dates of the last search for each database or the
period searched. Note any language or publication status restrictions (but refer
to Section 6.4.9). List individuals or organizations contacted.”
Reporting
See Cochrane Handbook http://www.cochrane-handbook.org/
Section 6.6
In methods: List all databases searched. Note the dates of the last search for each database AND the
period searched. Note any language or publication status restrictions List grey literature sources. List individuals or organizations contacted. List any journals and conference proceedings specifically
handsearched for the review. List any other sources searched (e.g. reference lists, the
internet).
Reporting
See IOM Standards http://iom.nationalacademies.org/Reports/2011/Finding-
What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx
In methods: Databases and other information sources used to identify
relevant studies Search strategy
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.
A researcher meets with you to discuss the search. You realize that she is pursuing the project individually. What are potential problems with this? How do you advise her?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.
You were brought in to a project to run the search. You completed this in Feb ‘14 and sent the project lead the results. In Dec ’14, you are given a manuscript, which has you listed as an author. You notice that the searches are not entirely yours and that you don’t like what they’ve done methodologically. How would you handle this situation?
RAPID FIRE SCENARIOS
In groups of 3, consider the following scenarios and identify solutions to these problems/issues that commonly arise in SR work.
Same scenario, but you notice that you made an error in the searches. What would you do?
Wrap Up & Other Stuff
Source: Liberati, 2009
• Eligibility screening, from PRISMA Explanations document
Wrap Up & Other Stuff
Types of Reporting Bias
Definition
Publication bias The publication or non-publication of research findings
Time lag bias The rapid or delayed publication of research findings
Multiple publication bias
The multiple or singular publication of research findings
Location bias The publication of research findings in journals with different ease of access or levels of indexing in standard databases
Citation bias The citation or non-citation of research findings
Language bias The publication of research findings in a particular language
Outcome reporting bias
The selective reporting of some outcomes but not others
Source: Cochrane Handbook, 10
Limitations
Wrap Up & Other Stuff
Source: Cochrane Handbook, 10
LimitationsTypes of Reporting Bias
Definition
Publication biasPositive results are over-represented in published literature. They are published and cited more frequently and published more quickly.
Negative results are less likely to be published and cited
What doctors don't know about the drugs they prescribe (Goldacre, TED Talk)
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
Wrap Up & Other Stuff
LimitationsTypes of Reporting Bias
Definition
Publication biasIndustry sponsored trials report “more favourable” outcomes when published (Ross 2009), but more industry-sponsored trials are left unpublished (Lundh 2012)
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
Wrap Up & Other Stuff
Source: Hopewell, S.
LimitationsTypes of Reporting Bias
Definition
Publication bias“Trials with positive results were published in approximately 4 to 5 years. Trials with null or negative results were published after about 6 to 8 years”
Our review shows that trials with positive results are published sooner than other trials.”
Hopewell S et al. Time to publication of results to clinical trials. Cochrane Review. PMID: 17443632
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
Wrap Up & Other Stuff
Source: Cochrane Handbook, 10.2.2.1
LimitationsTypes of Reporting Bias
Definition
Publication bias“It is not always obvious that multiple publications come from a single study, and one set of study participants may be included in an analysis twice. The inclusion of duplicated data may therefore lead to overestimation of intervention effects.”
Cochrane Handbook 10.2.2.1
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
Wrap Up & Other Stuff
Source: Cochrane Handbook, 10.2.2.2
LimitationsTypes of Reporting Bias
Definition
Publication biasSignificance of results and trial quality associated w/ journal impact.
“Choices regarding which databases to search may bias the effect estimate in a meta-analysis.”
Cochrane Handbook 10.2.2.2
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
• Strategy• Search multiple databases• Search subject-specific
resources• No access? Partner w/ other
librarians
Wrap Up & Other Stuff
Source: Cochrane Handbook, 10.2.2.2
LimitationsTypes of Reporting Bias
Definition
Publication biasBramer WM, et al. (2013) The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev. PMCID: PMC3882110.
Giustini D, et al. (2013) Google Scholar is not enough to be used alone for systematic reviews. Online J Public Health Inform. PMC3733758.
Sampson M, et al. (2003) Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. PMID: 14568625.
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
Wrap Up & Other Stuff
Source: Cochrane Handbook, 10.2.2.3
LimitationsTypes of Reporting Bias
Definition
Publication bias“Perusal of reference lists….is far from objective.”
Cochrane Handbook 10.2.2.3
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
• Strategy• Peruse reference lists (final
safety net)
Wrap Up & Other Stuff
Source: Cochrane Handbook, 10.2.2.4
LimitationsTypes of Reporting Bias
Definition
Publication bias“Reviews have often been exclusively based on studies published in English…. While the potential impact of [excluding non-English studies] may be minimal, it is difficult to predict in which cases this exclusion may bias a systematic review.”
Cochrane Handbook 10.2.2.4
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
• Strategy• Is it always feasible to include
non-English studies?• Case-by-case basis
Wrap Up & Other Stuff
Source: Cochrane Handbook, 10.2.2.5
LimitationsTypes of Reporting Bias
Definition
Publication bias“The choice of outcomes that are reported can be influenced by the results, potentially making published results misleading.”
Cochrane Handbook 10.2.2.5
Outcomes described in protocols are inconsistent with those reported in publications (Dwan K, 2013)
Time lag bias
Multiple publication bias
Location bias
Citation bias
Language bias
Outcome reporting bias
• Strategy• Develop a protocol. Not infallible
Wrap Up & Other Stuff
Search Strategy Considerations
Search multiple databases Cochrane Handbook suggests MEDLINE, Embase, and Cochrane Register
Include subject specific databases
Fewer limits MeSH termsScreen for eligibility manually instead
Language bias English?
Include grey literature sources
Trials registries / Conference proceedings / Protocols / Drug registries and other info / Contacting authors
AHRQ on Finding Grey Literature (Balshem, 2013)Cochrane Review on its value (Hopewell & MacDonald, 2007)
Librarian Involvement Librarian co-authors correlated higher quality search strategy reporting. (Rethlefsen 2015)
See also: Rethlefsen M. (2014). Engaging medical librarians to improve the quality of review articles. JAMA. PMID 25203078
Summary
Useful Resources
Expert Searching Listserv http://pss.mlanet.org/mailman/listinfo/
expertsearching_pss.mlanet.org Cochrane Handbook
http://handbook.cochrane.org/ PRISMA (remember the elaboration article)
http://www.prisma-statement.org/ Systematic Reviews: Opportunities for Librarians
http://www.lib.umich.edu/systematic-reviews-workshop
Acknowledgments
My colleagues involved in the Systematic Reviews: Opportunities for Librarians workshop. Specifically, Whitney Townsend at the University of Michigan
Joel Gagnier, Epidemiologist, University of Michigan,
MLA, for the Beyond the Search webinar series
References Abstrakr, http://abstrackr.cebm.brown.edu/ Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. (1992). A comparison of results of meta-analyses
of randomized control trials and recommendations of clinical experts: Treatments for myocardial infarction. JAMA, 268. 240-248. PMID 1535110.
Armstrong R, Jackson N, Doyle J, Waters E, Howes F. (2005). It's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health, 27(4).388-91. PMID 16311247.
Balshem H, Stevens A, Ansari M, Norris S, Kansagara D, Shamliyan T, Chou R, Chung M, Moher D, Dickersin K. (2013). Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program. PMID 24404628.
Beller EM, Chen JK, Wang UL, Glasziou PP. (2013). Are systematic reviews up-to-date at the time of publication? Syst Rev, 282. 36. PMID 23714302.
Benzies KM, Premji S, Hayden KA, Serrett K. (2006). State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs, 3(2). 55-61. PMID 17040510.
Bramer WM, Giustini D, Kramer BM, Anderson P. (2013). The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev, 23 (2). 115. PMCID PMC3882110.
Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions. http://handbook.cochrane.org/
Cochrane Community. Glossary. http://community.cochrane.org/glossary Cochrane Editorial Unit. Cochrane Editorial Unit, Methodological Expectations of Cochrane Intervention
Reviews. http://editorial-unit.cochrane.org/mecir Cochrane Review Group on HIV/AIDS. Data Extraction and Assessment Form.
http://hiv.cochrane.org/sites/hiv.cochrane.org/files/uploads/Data%20extraction%20form_RCTs.docx Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, McIlroy W, Oxman AD. (1993). Should
unpublished data be included in meta-analyses? Current convictions and controversies. JAMA, 269 (21). 2749–2753. PMID 8492400.
References Craane B, Dijkstra PU, Stappaerts K, De Laat A. (2012). Methodological quality of a systematic
review on physical therapy for temporomandibular disorders: influence of hand search and quality scales. Clin Oral Investig, 16 (1). 295-303. PMID 21128088.
Donahue KE, Gartlehner G, Jonas DE, Lux LJ, Thieda P, Jonas BL, Hansen RA, Morgan LC, Lohr KN. (2008). Systematic review: comparative effectiveness and harms of disease-modifying medications for rheumatoid arthritis. Ann Intern Med, 148 (2). 124-34. PMID 18025440.
Dwan K, Gamble C, Williamson PR, Kirkham JJ. (2013). Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One, 8 (7). e66844. PMID 23861749.
Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. (2003). How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess, 7 (1). 1-76. PMID 12583822.
EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews. http://www.ncbi.nlm.nih.gov/books/NBK274092
EPPI Reviewer 4. eppi.ioe.ac.uk/cms/er4/
Everett CR, Shah RV, Sehgal N, McKenzie-Brown AM. (2005). A systematic review of diagnostic utility of selective nerve root blocks. Pain Physician, 8 (2). 225-33. PMID 16850076.
Featherstone, R. Concept Map for Research Literature Reviews. http://www.slideshare.net/featherr/search-strategy-worksheet
Gagnier, JJ. (2015). Introduction to Systematic Reviews & Meta-Analyses. http://open.umich.edu/education/sph/epid757/summer2011/materials
Giustini D, Boulos MN. (2013). Google Scholar is not enough to be used alone for systematic reviews. Online J Public Health Inform. 5 (2). 214. PMCID PMC3733758
References Glanville J, Bayliss S, Booth A, Dundar Y, Fernandes H, Fleeman ND, Foster L, Fraser C, Fry-Smith A, Golder S,
Lefebvre C, Miller C, Paisley S, Payne L, Price A, Welch K. (2008). So many filters, so little time: the development of a search filter appraisal checklist. JMLA, 96(4). 356-61. PMID 18974813. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2568852/
Goldacre, B. (2012). What doctors don't know about the drugs they prescribe. TEDMED 2012. http://www.ted.com/talks/ben_goldacre_what_doctors_don_t_know_about_the_drugs_they_prescribe?language=en
Grant MJ, Booth A. (2009). A Typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 26 (2). 91-108. PMID 19490148.
Guyatt G, Rennie D, Meade MO, Cook, Deborah J. (2015). Summarizing the Evidence Worksheet. In: JAMAevidence, Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. 2nd Ed. http://www.jamaevidence.com/resource/520
HIRU, McMaster University. http://www.nlm.nih.gov/bsd/special_queries.html
HLWiki International. Software for systematic reviewing. http://hlwiki.slais.ubc.ca/index.php/Software_for_systematic_reviewing
Hopewell S, Clarke M, Lefebvre C, Scherer R. (2007). Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 18(2). MR000001. PMID 17443625.
Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M. (2002). A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med. 21 (11). 1625-34. PMID 12111923.
Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. (2007). Cochrane Database Syst Rev, 2. MR000011. PMID 17443632.
Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009, 1. MR000006. PMID 19160345.
References Hopewell S, McDonald S, Clarke M, Egger M. (2007). Grey literature in meta-analyses of randomized trials
of health care interventions. Cochrane Database Syst Rev. 18 (2). MR000010. PMID 17443631.
ICMJE. Defining the Role of Authors and Contributors. http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html
Institute of Medicine. Standards for Initiating a Systematic Review. Standards for Systematic Reviews. http://www.iom.edu/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx
InterTASC, http://www.sign.ac.uk/methodology/filters.html
Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. (2012). Evidence summaries: the evolution of a rapid review approach. Syst Rev, 1. 10. PMC3351736.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis P, Clarke M, Devereaux PJ, Kleijnen J, Moher D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine, 151 (4). W-65-W-94. PMID 19622512.
Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. (2012). Industry sponsorship and research outcome. Cochrane Database Syst Rev. 12. MR000033. PMID 23235689.
Mahood Q, Eerd DV, Irvin E. (2014). Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods, 5 (3). 221-34. PMID 26052848.
Martin JL, Pérez V, Sacristán M, Alvarez E. (2005). Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia. Eur Psychiatry, 20 (8). 550-3. PMID 15994063.
References McDonald S, Lefebvre C, Antes G, Galandi D, Gøtzsche P, Hammarquist C, Haugh M, Jensen KL, Kleijnen J,
Loep M, Pistotti V, Rüther A. (2002). The contribution of handsearching European general health care journals to the Cochrane Controlled Trials Register. Eval Health Prof, 25 (1), 65-75. PMID 11868446.
McKibbon KA, Wilczynski NL, Haynes RB, Hedges Team. (2009). Retrieving randomized controlled trials from medline: a comparison of 38 published search filters. Health Info Libr J, 26 (3). 187-202. PMID 19712211.
Nuti SV, Wayda B, Ranasinghe I, Wang S, Dreyer RP, Chen SI, Murugiah K. (2014). The use of google trends in health care research: a systematic review. PLoS One, 9 (10). e109583. PMID 25337815.
Oxman AD, Guyatt GH. (1993). The science of reviewing research. Annals of the New York Academy of Sciences, 703. 125-133. PMID 8192290.
PRISMA. http://www.prisma-statement.org/statement.htm
PROSPERO. Centre for Reviews and Dissemination. http://www.crd.york.ac.uk/PROSPERO/
PubMed Clinical Queries. http://www.ncbi.nlm.nih.gov/books/NBK3827/#pubmedhelp.Clinical_Queries_Filters
PubMed Special Queries. http://www.nlm.nih.gov/bsd/special_queries.html
Rethlefsen ML. (2013). I Want To Do a Systematic Review. Liblog: Newsletter of the Mayo Clinic Libraries. http://liblog.mayo.edu/2013/05/01/i-want-to-do-a-systematic-review/
Rethlefsen M. (2014). Engaging medical librarians to improve the quality of review articles. JAMA. 312 (10). 999-1000. PMID 25203078.
Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. (2015). Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol, 68 (6). 617-26. PMID: 25766056.
References Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. (2009). Trial publication after registration in
ClinicalTrials.Gov: a cross-sectional analysis. PLoS Med, 6 (9). e1000144. PMID 19901971.
Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. 2nd ed. Edinburgh: Churchill Livingstone, 2000.
Saleh AA, Ratajeski MA, Bertolet M. (2014). Grey Literature Searching for Health Sciences Systematic Reviews: A Prospective Study of Time Spent and Resources Utilized. Evid Based Libr Inf Pract, 9 (3). 28-50. PMID 25914722.
Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P. (2003) Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol, 56 (10). 943-55. PMID: 14568625
Sampson, M, McGowan J. (2006). Errors in search strategies were identified by type and frequency. Journal of Clinical Epidemiology, 59 (10). 1057-63. PMID 16980145.
Sampson M, Shojania KG, Garritty C, Horsley T, Ocampo M, Moher D. (2008). Systematic reviews can be produced and published faster. J Clin Epidemiol, 61 (6). 531-6. PMID 18471656.
SIGN. http://www.sign.ac.uk/methodology/filters.html
Sun G, Gaines J, Perla R, MacEachern M, Davis M, Shrank W. (2013). Systematic review of quality improvement initiatives in the intensive care setting. PROSPERO. http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42012003303#.VD6ZomNDR8F.
Sun GH, Raji CA, Maceachern MP, Burke JF. (2012). Olfactory identification testing as a predictor of the development of Alzheimer's dementia: a systematic review. Laryngoscope, 122 (7). 1455-62. PMID 22552846.
Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. (2014). Systematic review automation technologies. Syst Rev, 3. 74. PMID 25005128.