Searching for Relevant Studies

Searching for Relevant StudiesPrepared for:

The Agency for Healthcare Research and Quality (AHRQ)

Training Modules for Systematic Reviews Methods Guide

www.ahrq.gov

Systematic Review Process Overview

To substantiate why prudent literature searching is important

To describe important tenets of literature searching

To describe why transparency in the search process is important

Learning Objectives

Systematic reviews require comprehensive searches. Searching many databases by using a generic term gives

high recall but is very imprecise. For example: A July 23, 2009, Ovid search for “cancer.mp or neoplasms”

yielded 786,978 citations. That is too many citations to review to yield the 323 citations on

the role of tamoxifen and raloxifene in cancer, the reason for the search.

The goal is to carefully construct a search to keep precision high while maximizing recall.

Balancing Precision and Recall

Precision (specificity): the Precision (specificity): the proportion of retrieved articles proportion of retrieved articles

that are relevantthat are relevant

Recall (sensitivity): the Recall (sensitivity): the proportion of potentially relevant proportion of potentially relevant

articles retrieved by searcharticles retrieved by search

Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

One evaluation compared the results of popular studies against others with populations and methodologies that were similar or superior. Evaluation of 45 studies published from 1993 to 2000 in journals

with high impact factors that reported positive findings and were cited more than 1,000 times.

Results were compared against other studies with the same comparators by employing populations and methodologies that were similar or superior.

Opposite/null findings or much more dramatic effects were found 31 percent of the time. Eighty-three percent of nonrandomized studies and 23 percent

of randomized controlled trials were subsequently contradicted.

Presenting Popular Studies Is Not Enough

Ioannidis JP. JAMA 2005;294:218-28.

When a general topic is proposed, it is tempting to begin by extensively searching for primary literature.

Before performing the extensive literature search, it is crucial to understand the topic, devise an analytic framework, ask clearly defined key questions, and understand the scope of the review to be conducted.

Where To Begin

Although the topic area dictates the databases to be included, the following are common ones: MEDLINE (General) Cochrane Central (General) EMBASE (General, more international in scope) PsychLit/PsychINFO (Psychology) AIDSLine (HIV/AIDS) CINAHL (Nursing) TOXNET (Adverse Events/Toxicology)

Specialized Electronic Databases


Generally a minimum requirement for a thorough search of literature

Study 1 (Sassi et al., 2002): sensitivity and specificity of using MEDLINE versus nine databases This study focused on ability to retrieve economic analyses from

January to March 1997. MEDLINE-only searches had a sensitivity of 72 percent and a

specificity of 75 percent versus nine databases.

Study 2 (Betrán et al., 2005): systematic review of the prevalence of maternal mortality and morbidity from 1997 to 2002 Multiple databases were searched (MEDLINE, EMBASE, BIOSIS,

LILACS). 60 percent of citations were found in more than one database. The MEDLINE search had 20 percent of nonreplicated citations,

EMBASE had 7.4 percent, and LILACS had 5.6 percent.

MEDLINE Alone Is Not Enough

Sassi, F, et al. Med Care 2002;40:387-94; Betrán AP, et al. BMC Med Res Methodol 2005;5:6.

Perform a pilot search and compare to see if output matches articles already identified by the research group, key informants, and manual searches of references of these articles. Pilot searching is usually limited to MEDLINE and then

adapted to other databases. After pilot searching, refine the search. Searching the literature is an iterative process.

Identifying Key Articles To Refine a Pilot Search


Multiple searches within the same database may be necessary.

Efficacy and harms searches may be separate. If the available literature is small, a single broad search is

appropriate. If not so, separate searches are prudent.

Multiple Searches Help With anExtensive Literature Base (I)


An efficacy search may include the disease, intervention, and terms for randomized controlled trials (RCTs). Efficacy evaluations frequently are limited to RCTs.

Fewer inherent biases and higher quality evidence

A harms search may be broader and include just the intervention and harms and/or disease. The paucity of harms data in published RCTs frequently

requires that non-RCTs be included. Authors would likely be interested in harms regardless of

the reason for the intervention.

Multiple Searches Help With anExtensive Literature Base (II)

“OR” — makes the search broader “AND” — makes the search more selective “/” — means all subheadings are searched

When a search term is entered, you can selectively choose some subheadings (such as pharmacology for pharmacology studies).

Subheadings are generally imprecise; accepting all subheadings increases the yield.

“adj” — cuts down on miscellaneous citations “Heart adj failure” would find only instances in which “heart” is next to

“failure.” “$” — truncates a word with different endings

“Analy$” would find words like analysis, analyses, analyze, and analyse “.ti” or “.ab” — searches in the title or abstract for the word of interest

Can find citations where the word you search for is not a keyword “.mp” or multiple posting — a text word search; does not need to be a key

word

Boolean Operators Help ToBalance Precision and Recall

If the specific adverse effects of interest are specified in the key questions, they can be searched for directly (e.g., rhabdomyolysis, cancer).

General harms subheadings for MEDLINE would include: /adverse effects /poisoning /toxicity /chemically induced /contraindications /complications

General harms subheadings for EMBASE would include: /side effect /adverse drug reaction /drug toxicity /complication

Consider using a database dedicated to harms such as TOXNET (http://toxnet.nlm.nih.gov/).

Harms Searching


Filters have been developed and validated that can help balance precision and recall. Cochrane Collaboration http://www.cochrane-handbook.org

section 6.4.11 Search Filters, National Health Service Centre for Reviews and Dissemination

http://www.york.ac.uk/inst/crd/SysRev/!SSL!/WebHelp/SysRev3.htm

Scottish Intercollegiate Guidelines Network http://www.sign.ac.uk/methodology/filters.html

InterTASC Information Specialists’ Sub-Group http://www.york.ac.uk/inst/crd/intertasc/about.htm

Filters may not work in all circumstances

Filters or Hedges Are Validated Ways To Improve Precision and Recall


MEDLINE (Ovid) Search for Randomized Controlled Trials Using the Cochrane Highly Sensitive and Specific Search Strategy (Sensitivity and Precision Maximizing Version 2008)

1. randomized controlled trial.pt.

2. controlled clinical trial.pt.

3. randomized.ab.

4. placebo.ab.

5. clinical trials as topic.sh.

6. randomly.ab.

7. trial.ti.

8. 1 or 2 or 3 or 4 or 5 or 6 or 7

Example Filter (Hedge) for Finding Randomized Controlled Trials

Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

MEDLINE (Ovid) Search for Observational Studies Using the Scottish Intercollegiate Guidelines Network Observational Study Medline Search Filter

1. epidemiologic studies/

2. exp case control studies/

3. exp Cohort Studies/

4. case control.tw.

5. (cohort adj (study or studies)).tw.

6. cohort analy$.tw.

7. (follow up adj (study or studies)).tw.

8. (observational adj (study or studies)).tw.

9. longitudinal.tw.

10. retrospective.tw.

11. cross sectional.tw.

12. Cross-Sectional Studies/

13. 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12

Example Filter for Finding Nonrandomized Studies


Items to consider before hitting enter: Use of natural language terms (search as key word) Appropriate word truncation and spelling variation Appropriate use of limits such as language and years Appropriate use of searching for field, publication type, and

author

Peer Review of Electronic SearchStrategies (PRESS) Checklist (I)

Sampson M, et al. Sampson M, et al. PRESS: peer review of electronic search strategiesPRESS: peer review of electronic search strategies . 2008.. 2008.

Items to consider before hitting enter: Spelling errors/spelling in American and British English

(e.g., anaesthesia and anesthesia, analyse and analyze, oestrogen and estrogen)

Line errors that occur when searches are combined by using line numbers (e.g., “1 AND 3” instead of “1 AND 4”)

Appropriate use of Boolean operators Adaptation of search strategies for specific databases Are all appropriate headings used (refer to the thesaurus

for a specific database) Appropriate use of subheadings and floating subheadings

Peer Review of Electronic Search Strategies (PRESS) Checklist (II)

Sampson M, et al. Sampson M, et al. PRESS: peer review of electronic search strategiesPRESS: peer review of electronic search strategies . 2008.. 2008.

Publication bias is the tendency of certain types of trials (such as those with the largest effects) to be published.

Publication bias increases the risk that the observed effect might not reflect the true effect. It may negatively impact consistency, precision, and magnitude

of effect.

Expanding searches to include additional languages, citation tracking, hand searching, and grey literature can help identify and possibly minimize publication bias.

Considering Publication Bias WhenDeveloping a Search Strategy


Study (Turner et al., 2008): 74 antidepressant studies registered with the U.S. Food and Drug Administration 97 percent of positive studies published 39 percent of neutral or negative studies published

11 of 14 studies published in a way that conveyed the positive results but deemphasized the negative

When only published literature was meta-analyzed, a 32 percent increase in relative effect size occurred versus the more complete dataset of conducted trials.

Example of Publication Bias

Turner EH, et al. N Engl J Med 2008;358:252-60.

It is not a requirement to search non-English–language publications. Evidence-based Practice Centers should consider how

inclusion or exclusion of non-English–language studies might or might not bias the search.

It is usually advisable to include non-English– language studies when the evidence base may be highly restricted in English-language but broader in non-English–language studies. Such as, complementary and alternative medicine topics

Restricting Search by Languageand Impact on Publication Bias


Study 1 (Pham et al., 2008): evaluation of 42 systematic reviews, including 662 randomized controlled trials For Western medicine, language-restricted systematic

reviews did not introduce biased results when compared with language-inclusive ones, in terms of estimates of intervention effectiveness (risk odds ratio [ROR] = 1.02; 95% confidence interval [CI] = 0.83–1.26).

For complementary and alternative medicine, however, language-restricted systematic reviews resulted in a 63 percent smaller protective effect estimate when compared with language-inclusive reviews (ROR = 1.63; 95% CI = 1.03–2.60).

English Only or All Languages (I)

Pham B, et al. J Clin Epidemiol 2005;58:769-76.

Study 2 (Vickers et al., 1998): evaluation of acupuncture trials conducted in England versus China, Japan, Russia, and Taiwan Results of randomized controlled trials or controlled clinical

trials in England were favorable 75 percent of the time. Results in China, Japan, Russia, and Taiwan were favorable

99, 89, 97, and 95 percent of the time, respectively. Researchers should carefully consider how to manage

data from these countries.

English Only or All Languages (II)

Vickers A, et al. Control Clin Trials 1998;19:159-66.

Forward citation tracking: databases that provide information on articles cited by other authors This method relies on the author’s choice to cite an article,

rather than key words or indexing. Can turn up unique items and can complement a

traditional database search. Databases currently in use are Web of Science, Scopus,

Google Scholar, PubReMiner, and PubFocus. Web of Science and Scopus have access fees but have the

most developed search and export interfaces. Google Scholar, PuReMiner, and PubFocus are free access

but are not as developed.

Citation Tracking (I)


Backward citation tracing: reading references of key articles New search terms, poorly indexed journals, or inappropriate

limiters in the original search can be identified

“Related article” links: useful when designing and refining a search but not for a formal search Difficult to perform systematically, report strategy with

transparency, and reproduce the search

Citation Tracking (II)


Not all journals or sources can be indexed in databases. Some journals index only certain article types. Abstracts, conference proceedings, and supplements may

not be indexed.

Backward citation tracking and key informant discussion can identify these journals.

Ulrich has a useful Web site to identify journals in a specific topic area (available at: http://www.ulrichsweb.com/ulrichsweb/). If Ulrich indicates an important journal is not indexed fully

in databases used, search the journal by hand.

Hand Searching To Reduce Publication Bias


Study 1 (Hopewell et al., 2007): using MEDLINE to search 22 specialized journals to identify randomized controlled trials (RCTs) was compared with hand searching of the same journals 117 of the 462 RCTs (25%) were not retrieved by MEDLINE

because of a lack of the publication-type terms “randomized controlled trial” or “controlled clinical trial.” Additional articles were missed in journals without full journal

indexing.

Study 2 (Jadad et al., 1993): using MEDLINE to search nine pain and anesthesia journals was compared with hand searching 13 percent of eligible articles were not retrieved with MEDLINE

alone.

Benefits of Hand Searching

Hopewell S, et al. Cochrane Database Syst Rev 2007;(2):MR000010;Jadad AR, McQuay HJ. Online J Curr Clin Trials 1993;(33).

Grey literature: literature produced at all levels of government, academics, business, and industry that is not controlled by commercial publishers Usually not systematically identified, stored, or indexed Can help overcome publication bias

Least efficient type of literature to search and may be (or be perceived to be) of lower quality Useful in areas with little published evidence, in fields that are

rapidly changing (e.g., devices, surgery), or in fields that are highly interdisciplinary (e.g., mental health, alternative medicine)

Using Grey LiteratureTo Reduce Publication Bias


Study (Hopewell et al., 2008): reviewed five systematic reviews conducted to evaluate the impact of grey literature in meta-analyses of randomized controlled trials All systematic reviews found more positive results with

published literature than with grey literature (risk odds ratio = 1.09; 95% confidence interval = 1.03–1.16)

Grey Literature

Hopewell S, et al. Cochrane Database Syst Rev 2007;(2):MR000010.

The approval process for new drugs and devices involves submission of data that may not be published or may be incompletely published elsewhere Good check for publication bias

Regulatory Web sites: Drugs@FDA; http://www.accessdata.fda.gov/Scripts/cder/DrugsatFDA/

index.cfm?fuseaction=Search.search_Drug_Name FDA Center for Drug Evaluation and Research (CDER) Archives:

http://www.fda.gov/cder/archives/default.htm Dockets for FDA: http://www.fda.gov/ohrms/dockets/default.htm Devices@FDA: http://www.accessdata.fda.gov/Scripts/cdrh/devicesatFDA/ Health Canada Drug Product Search:

http://www.hc-sc.gc.ca/dhp-mps/prodpharma/databasdon/index-eng.php European Medicines Agencies European Public Assessment Reports for

Authorized Medicinal Products for Human Use: http://www.emea.europa.eu/htms/human/epar/a.htm

Grey Literature: Regulatory Search


Online registries include trial name, type, and brief methodology May contain results of completed but unpublished trials

Registry Web sites: ClinicalTrials.gov: http://clinicaltrials.gov/ct2/search ClinicalStudyResults.org:

http://www.clinicalstudyresults.org/home/ Current Controlled Trials: http://www.controlled-

trials.com/mrct/ Australian New Zealand Clinical Trials Registry:

http://www.anzctr.org.au/trialSearch.aspx Netherlands Trial Registry: http://www.trialregister.nl

Grey Literature: Online Registries


Much grey literature is unlikely to be published.

When published, final data often differ from those in the conference abstract.

Use key informants to identify the most important meetings in the field of interest and hand search the abstracts and conference proceedings in journal supplements. Some databases are also available: ProceedingsFirst;

Biological Abstracts; Conference Papers Index; Conference Proceedings Citation Index (part of the Web of Science); and BioWizard Scholar Conference Abstract Search.

Grey Literature: Abstractsand Conference Proceedings


Theses and dissertations are becoming increasingly available as institutions post them online.

Databases: ProQuest Dissertation and Theses

The most comprehensive collection in the United States, with full text of dissertations through their Digital Archiving and Access program

Index to Theses in Great Britain and Ireland Networked Digital Library of Theses and Dissertations

(NDLTD) Dissertation Abstracts Online

Grey Literature: Theses and Dissertations


Aside from governmental regulatory sites, other governmental agencies support or collect grey literature. Computer Retrieval of Information on Scientific Projects

(CRISP): http://crisp.cit.nih.gov Searchable database of federally funded biomedical research

Health Services Research Projects in Progress (HSRPROJ): http://wwwcf.nlm.nih.gov/hsr_project/home_proj.cfm Database of ongoing grants and projects in health services

research Science Accelerator: http://www.scienceaccelerator.gov/

From the Office of Scientific and Technical Information and the U.S. Department of Energy

Defense Technical Information Center (DTIC): http://www.dtic.mil/dtic/ From the U.S. Department of Defense, including the military

health system

Grey Literature: Government Documents


Grey Matters: http://www.cadth.ca/index.php/en/ cadth/products A practical tool for evidence-based searching Collated list of sources for grey literature appropriate for health

sciences by the Canadian Agency for Drugs and Technology in Health (CADTH)

System for Information on Grey Literature in Europe (SIGLE): http://opensigle.inist.fr/ Provides bibliographic references to grey literature in Europe

New York Academy of Medicine (NYAM): http://www.nyam.org/library.grey.shtml Provides NYAM grey literature with a list of other organizations

that do the same

Grey Literature: General Sources


Researchers can be contacted to determine whether or not projects in the grey literature are published in a peer-reviewed form.

Prolific researchers in the field can be contacted to see if they know of any citations that were missed by the literature search.

Search engine (e.g., Yahoo, Google) queries with the individual’s name or institution can be a good place to start. Contact information in manuscripts is also a reasonable

place to start.

Directly Contacting Researchers


SIPs are provided by the company manufacturing the drug or device.

SIPs contain information about products, including the product label and published and unpublished trials or studies. They: can help to overcome publication bias by identifying trials

that remain unpublished; can identify soon-to-be-released trials that can ultimately

be captured in comparative effectiveness reviews (CERs); and

allow drug and device manufacturers to be explicitly involved in helping provide a literature base to a CER.

Scientific Information Packets (SIPs)


Scientific Information Packets


Accurate and transparent reporting of search strategies is important. Searches need to be updated toward the conclusion of a

systematic review to ensure that no new information has come out in the intervening period.

Stakeholders and end-users want assurances that the systematic review was free of bias and that adequate search principles were followed.

Searches will need to be rerun if the systematic review is updated several years after it is completed.

Documenting the search strategy so that a skilled literature searcher can reproduce it is very important.

Reporting Search Strategies: Transparency


Transparent Reporting

Databases used

Dates covered

Search terms

Language restrictions

Nondatabase methods used

Inclusion/exclusion criteria

Full electronic search strategy

Publication-related restrictions

End date of search

List of excluded references

Qualifications of searcher

Number of references identified

QUOROM - or PRISMA-style flow diagram accounting for all references

Evidence of search effectiveness

Statement of filters employed

Description of sampling strategyCampbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

Transparent Reporting: QUOROM Figure


Transparent Reporting: PRISMA Figure

Included

Screenin

gEligibility

Identification

The next few slides present a comparison of a controlled trial search and an observational study search.

See how the search pieces are created and then come together. Note the use of Boolean operators.

Example of Two Searches

Example of Controlled Trial Search (I)


Example of Controlled Trial Search (II)


Example of Controlled Trial Search (III)

Coleman CI, et al. Comparative Effectiveness Review No. 18. Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

Example of Observational Study Search (I)


Example of Observational Study Search (II)


Example of Observational Study Search (III)


Searching is important: balance precision and recall. MEDLINE is not enough.

Multiple strategies improve completeness of search.

Filters are a good place to start. More than one search may be needed (e.g.,

separately for “efficacy” and “harms”). Report strategy with transparency.

Key Messages

Betran AP, Say L, Gulmezoglu AM, et al. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol 2005 ;5:6.

Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.

Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease. Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

Hopewell S, Clarke M, Lusher A, et al. A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med 2002;21:1625-34.

References (I)

Hopewell S, McDonald S, Clarke MJ, et al. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007;(2):MR000010.

Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005;294:218-28.

Jadad AR, McQuay HJ. A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J Curr Clin Trials 1993;(33).

Pham B, Klassen TP, Lawson ML, et al. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol 2005;58:769-76.

References (II)

Sampson M, McGowan J, Lefebvre C, et al. PRESS: peer review of electronic search strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008.

Sassi F, Archard L, McDaid D. Searching literature databases for health care economic evaluations: how systematic can we afford to be? Med Care 2002;40:387-94.

Turner EH, Matthews AM, Linardatos E. et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252-60.

Vickers A, Goyal N, Harland R, et al. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials 1998;19:159-66.

References (III)

This module was prepared by C. Michael White, Pharm.D., FCP, FCCP, a member of the University of Connecticut/Hartford Hospital Evidence-based Practice Center.

This module is based on chapter 5 in version 1.0 of the Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews (available at: http://effectivehealthcare.ahrq.gov/repFiles/2007_10Draft MethodsGuide.pdf).

Author

Searching for Relevant Studies

Documents

Transcript of Searching for Relevant Studies