The AHRQ Training Modules forthe Systematic Reviews Methods
Guide:An Introduction
Prepared for:
The Agency for Healthcare Research and Quality (AHRQ)
Training Modules for Systematic Reviews Methods Guide
www.ahrq.gov
To review the goals and activities of the Evidence-based Practice Center (EPC) program in relation to the Effective Health Care (EHC) Program
To describe the roles and responsibilities of the different participants in the EHC Program in relation to the EPC program
To outline the purpose and components of the Training Modules for the Methods Guide for Comparative Effectiveness Reviews (CERs)
To define important terms and concepts used in the Training Modules for the Methods Guide for CERs
Learning Objectives
Section 1013 of the 2003 Medicare Prescription Drug, Improvement, and Modernization Act authorized the Agency for Healthcare Research and Quality (AHRQ) to improve the quality, effectiveness, and efficiency of health care delivered through Medicare, Medicaid, and State Children’s Health Insurance programs.
AHRQ started the Effective Health Care Program in 2005 upon appropriation of funds.
In 2009, the American Recovery and Reinvestment Act (ARRA) awarded additional funding to AHRQ to:
broaden comparative effectiveness research,
build an infrastructure to support the expanded efforts, and
invest in innovative mechanisms to disseminate findings of research to health care decisionmakers.
What Is the Effective Health Care Program?
Conducts effectiveness and comparative effectiveness research to inform the health care decisions of patients, providers, and policymakers Involves individual researchers, research centers, and
academic organizations in the United States and Canada Comparative effectiveness research includes:
Reviews and syntheses of published and unpublished scientific evidence
New scientific evidence and analytic tools
Translates research findings into easily understood formats for patients, providers, and policymakers
What Does the Effective Health Care Program Do?
The conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat, and monitor health conditions in “real world” settings.
The purpose of this research is to improve health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decisionmakers and responding to their expressed needs about which interventions are most effective for which patients under specific circumstances.
What Is Comparative Effectiveness Research?
Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. June 30, 2009. Available at: http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf.
Answer real questions Health care decisionmaking
Best evidence Internal and external validity
Comparative Real-world comparisons
Effectiveness Patient-centered outcomes of interest
The ABCs of Comparative Effectiveness Research
Relevant and timely Useful for decisionmakers Stakeholder input for topic nomination and development
Objective and scientifically rigorous Identification of methodological standards Careful consideration of potential conflicts of interest Ensure balanced approach to topic and research
Open for public participation and transparent Accountable Clear documentation of processes Stakeholder input for topic nomination, development, and
review of draft reports
Effective Health Care Program Principles
Slutsky J, et al. Comparing medical interventions: AHRQ and the Effective Health Care Program. In: Methods guide for comparative effectiveness reviews. Available at: http://effectivehealthcare. ahrq.gov/ehc/products/60/318/2008_1118CompareInterventions.pdf.
Components of theEffective Health Care Program
Stakeholder Input and Involvement
1.Evidence Synthesis Research reviews summarize the scientific literature on comparative
effectiveness questions and highlight areas for future research. Comparative effectiveness and effectiveness reviews outline the effectiveness, benefits,
and harms of treatment options.
Technical briefs describe the state of the literature for new or emerging health care tests or treatments.
2.Evidence Generation New research reports analyze new data to answer comparative effectiveness
questions. Research on health care databases may be supported through the Effective Health Care
Program, specifically through the DEcIDE Network or the CERTs.
Clinical comparative effectiveness trials will be supported through extramural grants.
3.Dissemination and Translation Summary Guides translate findings from research reviews for patients,
providers, or policymakers.
How Does the Effective Health Care Program Answer Real-World Questions?
Evidence-based Practice Centers (EPCs) The John M. Eisenberg Center for Clinical Decisions and
Communications Science The Scientific Research Center (SRC) Partners for new research:
Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Network The Centers for Education and Research on Therapeutics (CERTS) Extramural grantees and other researchers
Stakeholders: Formal Stakeholder Group Key Informants Technical experts Public input
Task order officers at the Agency for Healthcare Research and Quality
Who Are the Key Playersin the Effective Health Care Program?
Each Player on the Team Has a Different Role
Patients, health care providers, and policymakers make better health care decisions
StakeholdersNominate topics, provide input on
questions, reviews, and
program
DEcIDE, CERTs, GranteesEvidence Generation
Evidence-based Practice Centers Evidence Synthesis and
Evidence Need Identification
Eisenberg Center Evidence Translation and Dissemination
Scientific Resource Centercoordinates topic selection/peer review/public input,
provides research support
AHRQ Task Order OfficersDevelop, coordinate, and monitor
The Game Plan
Evidence-based Practice Center researchers directly engage stakeholders for specific topics or reports Specific input through focus groups, key informant panels
Topic development and refinement Determining Research needs
Public input through posting and comment periods Draft Key Questions Draft research reviews
Effective Health Care (EHC) Program members engage stakeholders for the Program as a whole Open Web topic nominations Citizen’s forum (in development) Stakeholder groups provide input on EHC Program processes
Volunteer members for a 2-year term Representatives of various stakeholder groups
Engaging Stakeholders
Points of Stakeholder Engagementfor Systematic Reviews
Topic Generation
Topic Development
Topic Refinement
Research Review
Research Needs Development
Report Translation and Dissemination
Generate nominations
relevant to “real-world” health care
decisions
Clarify nomination and decisional
dilemma
Clarify and refine key questions to accurately reflect
the decisional issue
Provide clinical & methodological input to EPC’s
protocol
Provide input to develop and
prioritize research gaps
Test and provide input on summary
guide development
Evidence-based Practice Centers Eisenberg Center andAHRQ OCKT
Web-based and SRC Stakeholder
Engagement Team
Answer real-world health
questions
Ensure appropriate context is
considered
Ensure key questions accurately
reflect decisional dilemmas
Provide clinical &
methodological input
Craft meaningful
messages and disseminate
information to those who
need it
Develop and prioritize
research gaps
AHRQ Effective Health Care ProgramPoints of Stakeholder Engagement for Systematic Reviews
Pro
gra
mP
ha
seE
ng
ag
em
en
t P
urp
ose
Sta
keh
old
er
Inte
rest
AHRQ = Agency for Healthcare Research and Quality; OCKT = Office of Communications and Knowledge Transfer; SRC = Scientific Resource Center
Research reviews completed under the Effective Health Care and Generalist Programs are broad and balanced in scope: Useful to a broad range of stakeholders Scope determined by the Evidence-based Practice Center through topic development
Comparative effectiveness reviews Effectiveness reviews Technical briefs
Reviews requested by Federal partners may have specific purpose and scope as defined by the partner: Center for Medicare and Medicaid Services (CMS), through the Technology
Assessment Program U.S. Preventive Services Task Force (USPSTF) National Institutes of Health (NIH), through the Office of Medical Applications of
Research (OMAR) Other federal partners: AHRQ, Centers for Disease Control and Prevention (CDC),
Food and Drug Administration (FDA), and other NIH centers
Evidence-based Practice Centers Conduct Research Reviews for a Variety of Purposes
The Evidence-based Practice Center program was instituted in 1997. The following are the centers that were awarded contracts from the Agency for Healthcare Research and Quality in 2007. The contracts are competitive on a 5-year cycle. Blue Cross and Blue Shield Association, Technology Evaluation Center, Chicago, IL Duke University, Durham, NC ECRI Institute, Plymouth Meeting, PA Johns Hopkins University, Baltimore, MD McMaster University, Hamilton, Ontario, Canada Minnesota Evidence-based Practice Center, Minneapolis, MN Oregon Evidence-based Practice Center, Portland, OR RAND Corporation, Santa Monica, CA RTI International-University of North Carolina, Research Triangle, NC Tufts Medical Center, Boston, MA University of Alberta, Edmonton, Alberta, Canada University of Connecticut, Storrs, CT University of Ottawa, Ottawa, Ontario, Canada Vanderbilt University Medical Center, Nashville, TN
The Current Evidence-based Practice Centers
The Scientific Resource Center (SRC) is currently* located at the Oregon Evidence-based Practice Center (EPC) and supports the activities of the Effective Health Care (EHC) Program by: Communicating with stakeholders and assisting EPC members to
ensure that EHC Program products meet the practical needs of providers, patients, and policymakers
Receiving and sorting topic nominations Facilitating and coordinating the decisionmaking process for
topic selection of research reviews Coordinating peer review and public input for research reviews Collaborating with EPC members to develop consistent and
rigorous scientific methods in conducting research reviews.
The Scientific Resource Center
*Awarded by competitive bid in 2007.
The John M. Eisenberg Center for Clinical Decisions and Communications Science is currently* located at Baylor College of Medicine in Houston, Texas.
The Center works with products developed by Evidence-based Practice Center members to translate research reviews into short, easy-to-read guides that can be used by consumers, clinicians, and policymakers.
Involvement of the Center begins at topic development and continues until the guides are disseminated, generally within 6 months of the final research review. Early involvement helps to: improve the usefulness of research products, and optimize fidelity of translation.
The John M. Eisenberg Center for Clinical Decisions and Communications Science
*Awarded by competitive bid in 2008.
Serve as the contracting officer technical representative for all Effective Health Care Program partners Specific task order officer listed for each task order Contact the task order officer for questions regarding:
Task order Collaboration or coordination with other partners in the EHC and Evidence-
based Practice Center Programs
Participate in calls and meetings for each research review step, from topic development through publication
Provide input on scientific, protocol, and process issues Help EPC researchers identify opportunities for
collaboration within the EHC Program to address common challenges and communicate best practices
Agency for Healthcare Researchand Quality Task Order Officers
In 2009, eight Evidence-based Practice Centers were awarded additional task orders to:
Develop topics for potential research review within a given area of concentration that corresponds to AHRQ's priority conditions or cross-cutting themes
Conduct research reviews within a given area of concentration
Identify important future research that is needed to fill gaps and help patients, physicians, and policymakers make health care decisions within a given area of concentration Develop methods for rigorous and transparent methods for determining
specific study design and prioritizing research needs Involve stakeholders, which include funders of new research and
investigators who conduct new research
American Recovery and Reinvestment Act–Funded Evidence-based Practice Center Activities
The eight Evidence-based Practice Centers (EPCs) that were awarded funding for 3 years through the American Recovery and Reinvestment Act (ARRA) in 2009 will focus on the following research areas:
Arthritis, functional limitations, and disability Minnesota EPC
Cancer and infectious disease Blue Cross and Blue Shield Association EPC
Cardiovascular and pulmonary disease Duke University EPC
Mental health RTI International-University of North Carolina EPC
Pregnancy, developmental delays, and autism Vanderbilt University EPC
Diabetes, obesity, and peptic ulcer disease Johns Hopkins University EPC
Health care delivery, prevention, and behavioral intervention Oregon EPC
Diagnostics and devices Tufts Medical Center EPC
Areas of Concentration for the ARRA-Funded Evidence-based Practice Centers
Evidence-based Practice Centers work with researchers, funders, and other stakeholders to determine important research needs:
The DEcIDE Network An AHRQ contract-funded network of research centers, primarily organized by
consortia around selected priority conditions Generates new comparative effectiveness evidence through analysis of existing
health care databases and patient registries, and new data collection
Centers for Education and Research on Therapeutics (CERTs) A network of research centers funded by AHRQ as cooperative agreements, each
addressing different themes, including diseases, populations, data systems, and methodologies
Generate, translate, and disseminate new evidence on benefits and harms of therapeutics
AHRQ extramural grantees AHRQ grant-funded researchers Conduct clinical comparative effectiveness studies
Partners for New Research
AHRQ = Agency for Healthcare Research and QualityDEcIDE = Developing Evidence to Inform Decisions about Effectiveness
Acumen, LLC, Burlingame, CA Brigham and Women's Hospital, Boston, MA Duke University, Durham, NC Harvard Pilgrim Health Care, Harvard Medical School, Boston, MA Johns Hopkins University, Baltimore, MD Outcome Sciences, Cambridge, MA RTI International, Research Triangle Park, NC University of Colorado at Denver and Health Sciences Center, Aurora, CO University of Illinois at Chicago, Chicago, IL University of Maryland at Baltimore, Baltimore, MD University of North Carolina at Chapel Hill, Chapel Hill, NC University of Pennsylvania School of Medicine, Philadelphia, PA Vanderbilt University Medical Center, Nashville, TN
The Current DEcIDE Network Centers*
*Awarded by competitive bid in 2005DEcIDE = Developing Evidence to Inform Decisions about Effectiveness
Clinical and economic issues in hospital settings — University of Chicago Consumer education and patient adherence — University of Texas M.D. Anderson
Cancer Center and Baylor College of Medicine Therapies for heart and blood vessel disorders — Duke University Medical Center Musculoskeletal disorders — University of Alabama Drug interactions, women's health — University of Arizona/The Critical Path
Institute Drug use, safety, and effectiveness in managed care — HMO Research Network Health information technology — Brigham and Women's Hospital Anti-infective therapeutics use and resistance — University of Pennsylvania
School of Medicine Therapeutic medical devices — Weill Medical College of Cornell University Mental health therapeutics — Rutgers, The State University of New Jersey Therapies for older adults and the effects of aging — University of Iowa Pediatric therapeutics — Cincinnati's Children's Hospital Medical Center Tools for optimizing prescribing — University of Illinois at Chicago Therapeutics for vulnerable populations — Vanderbilt University Medical Center
The Current CERTs Centers*
*Grants awarded by competitive bid in 2006CERTS = Centers for Education and Research on Therapeutics
The purpose of the training modules is: To provide practical training on the Methods Guide for Conducting
Comparative Effectiveness Reviews
The Methods Guide is: Intended to improve the transparency, consistency, and scientific rigor of the
Effective Health Care (EHC) Program A collaborative effort of AHRQ, the Scientific Resource Center (SRC), and the
Evidence-based Practice Centers (EPCs) A living document with revisions as new evidence and understanding grows
(initial draft posted in 2007) Posted on the EHC Program Web site
Other materials available to new EPC investigators or staff: The Methods Guide Stakeholder training (soon to be available to EPCs) Program materials, such as guides and templates (available to EPCs on
secure site) Learning Network (soon to be available to EPCs on secure site) Methods library and weekly e-mail methods article alert (contact task order
officer or SRC to be added to the listserv)
AHRQ Training Modules forthe Systematic Reviews Methods Guide
1. Topic Refinement2. Analytic Frameworks3. Study Eligibility Criteria4. Searching for Relevant Studies5. When To Select Observational Studies as Evidence 6. Data Extraction7. Assessing the Quality of Individual Studies8. Assessing Applicability9. Presentation of Findings10. Quantitative Synthesis (I)11. Quantitative Synthesis (II)12. Grading Strength of Evidence13. Reporting the Review
The Training Modules
Systematic Review Process Overview
Prepare Prepare TopicTopic
• Refine Refine topictopic
•Develop Develop analytic analytic frameworkframework
Search for and Search for and Select Studies for Select Studies for
InclusionInclusion
•Identify study Identify study eligibility criteriaeligibility criteria
•Search for Search for relevant studiesrelevant studies
•Select evidence Select evidence for inclusionfor inclusion
Extract Extract Data from Data from StudiesStudies
Analyze and Analyze and Synthesize Synthesize
StudiesStudies
•Assess the quality Assess the quality of individual studiesof individual studies
•Assess Assess applicabilityapplicability
•Present findingsPresent findings
•Synthesize Synthesize quantitative dataquantitative data
•Grade strength of Grade strength of evidenceevidence
Report Report Systematic Systematic
ReviewReview
Identify gaps and future research needsIdentify gaps and future research needs
Important Terms and Concepts (I)
Research Review or Evidence SynthesisResearch Review or Evidence Synthesis
Systematic Review Systematic Review
Comparative Effectiveness ReviewComparative Effectiveness Review Technical BriefTechnical Brief
Effectiveness ReviewEffectiveness Review
PICO ≈ PICO(TS) ≈ PICOTS ≈ PICOT ≈ PICOS Population Intervention Comparison Outcome Timing of outcome assessment Setting (sometimes study design)
Important Terms and Concepts (II)
www.effectivehealthcare.ahrq.gov Public Web site of the Effective Health Care (EHC) Program:
EHC Program published methods papers, including the Methods Guide EHC Program published research reviews and summary guides Opportunities for collaborations through listserv notifications Key question, draft report, and final report postings
https://www.kpchr.org/ehc/system/login.aspx Secure site available only to Evidence-based Practice Center (EPC) investigators Contact task order officer or Scientific Resource Center for access Enable “remember me” function to improve usability and access
Policy and process documents, including templates and topic development working documents
Learning Network — a forum for asking questions and sharing experiences and ideas
Contact information for EHC Program and partners http://mc.manuscriptcentral.com/ehc
Scholar One Manuscript Central — peer-review management system (contact task order officer or the Scientific Resource Center for access)
Upload draft and final documents through this site
Helpful Web Sites forEvidence-based Practice Center Investigators
Effective Health Care Program. Methods guide for effectiveness and comparative effectiveness reviews. Rockville, MD: Agency for Health Care Research and Quality, 2009. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318.
Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Washington, DC: U.S. Department of Health and Human Services; June 30, 2009. Available at: http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf.
Helfand M, Balshem H. Principles in developing and applying guidance. In: Methods Reference Guide for Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality, Posted August 2009. Available at: http://effectivehealthcare.ahrq.gov/healthInfo. cfm?infotype=rr&ProcessID=60.
References
Slutsky J, Atkins D, Chang S, et al. Comparing medical interventions: AHRQ and the Effective Health Care Program. In: Methods guide for comparative effectiveness reviews. Rockville, MD: Agency for Health Care Research and Quality, Posted November 2008. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2008_1118CompareInterventions.pdf.
Slutsky J, Atkins D, Chang S, et al. AHRQ series paper 1: comparing medical interventions: AHRQ and the Effective Health-Care Program. J Clin Epidemiol 2010;63:481-483.
References
This presentation was prepared by Stephanie Chang, M.D., M.P.H., and Elisabeth Kato, M.D., M.R.P., at the Agency for Healthcare Research and Quality (AHRQ) Center for Outcomes and Evidence.
Authors
Top Related