DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO...
Transcript of DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO...
![Page 1: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/1.jpg)
dcasip.medicine.duke.edu
DEVELOPING AND EVALUATING YOUR INTERVENTIONREBEKAH MOEHRING, MD, MPH
SHEA ANTIMICROBIAL STEWARDSHIP RESEARCH WORKSHOP
NOVEMBER 16, 2017; CHICAGO, IL
![Page 2: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/2.jpg)
DisclosuresGrants: CDC, CDC Foundation, AHRQRoyalties: UpToDate, Inc.Honoraria: SHEA (Merck grant to SHEA)
Borrowed Slides: Tamar Barlam; Julie Szymczak
![Page 3: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/3.jpg)
Objective: Develop and Evaluate InterventionsOutline key aspects of stewardship intervention design to consider before implementation to optimize research successDescribe implementation strategies that will aid in more systematic assessment of stewardship initiatives
![Page 4: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/4.jpg)
Goals for Stewardship Researchers
Provide a guideline that diverse stakeholders find usefulMore detailed, implementation-oriented focus compared with prior guidelinesUse the GRADE system to rank the guideline’s recommendations and the level of evidence
My work here…Slide: Dr. Tamar Barlam, Boston Univ
![Page 5: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/5.jpg)
GRADE and PICODevelop PICO questions to frame topicsPopulation of interestIntervention or indicatorComparator or control groupOutcome
Slide: Dr. Tamar Barlam, Boston Univ
![Page 6: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/6.jpg)
From evidence to recommendations
RCTObser-vationalstudy
High level recommen-
dation
Lower level recommen-
dation
Old systemSlide: Dr. Yngve Falck-Ytter and Dr. Tamar Barlam
Quality of evidence
Balance between benefits, harms & burdens
Patients’ values &
preferences
Grading of Recommendations Assessment, Development and
Evaluation (GRADE)
![Page 7: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/7.jpg)
GRADEGRADE: all evidence may be examinedIf there is a question, then there is evidenceLack of RCTs does not mean weak evidenceHigher quality indirect data may be preferable than low quality direct data
Slide: Dr. Yngve Falck-Ytter and Dr. Tamar Barlam
Quality of Evidence
![Page 8: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/8.jpg)
Your Charge…Research that can help inform stewardship practiceQualitative and quantitative implementation scientific inquiryHopefully, the next revision of this guideline will have many more strong recommendations as the quality of the evidence improves
Slide: Dr. Tamar Barlam, Boston Univ
![Page 9: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/9.jpg)
Homework: Read ICHE
Topics, published in ICHE 2016 Randomized Controlled Trials Quasi-experimental Designs
Use of Administrative and Surveillance Databases Survey and Qualitative Research
Observational Studies Mathematical Modeling
![Page 10: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/10.jpg)
What does it mean to intervene?
![Page 11: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/11.jpg)
What does it mean to intervene in antimicrobial stewardship?
![Page 12: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/12.jpg)
InnovationThe best ideas are the most simple and easy to explainPlausibilityAdaptation from another field or prior studyTips:Talk it through with your team and collaboratorsKeep a running list
A form of cooperative argumentative dialogue between individuals, based on asking and answering questions to stimulate critical
thinking and to draw out ideas and underlying presumptions.
![Page 13: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/13.jpg)
Nosce te ipsum
![Page 14: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/14.jpg)
The Scientific Method
![Page 15: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/15.jpg)
Nosce te ipsum: What are your goals?Intervention development or PilotEfficacy and effectiveness studiesImplementation studies
Will it float? Can it be applied?Can it work?
![Page 16: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/16.jpg)
Nosce te ipsum: What are your goals?
Duke University Medical Center Library and Archives. http://guides.mclibrary.duke.edu/ebm/studydesign Accessed 11-1-2016.
Establish causality
Observe Associations
Generate Hypothesis
Observational Studies:Cohort Studies
Case Control StudiesCross-Sectional Studies
Ecologic studiesCase Series
Experimental Studies:Randomized Controlled
TrialsQuasi-experimental
Studies
![Page 17: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/17.jpg)
Nosce te ipsum: What is the targeted process and setting?ObserveSocio-behavioral factorsReview problem and possible interventions with front-line stakeholders Understand operational issues
![Page 18: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/18.jpg)
Nosce te ipsum: What is the targeted process and setting?ObserveSocio-behavioral factorsReview problem and possible interventions with front-line stakeholders Understand operational issues
![Page 19: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/19.jpg)
Process: Anticipate setbacks, elicit feedbackHow burdensome would the proposed change in process be? What can you do to minimize that burden?Allow some flexibilityHow will data collection fit in?Careful research, revision, and vetting of the “products” and data collection tools
Redundant Events by spectrum group
Anti-pseudomonal 220 (38) 1010 4 (2-6)
Anti-anaerobe 176 (30) 668 3 (2-5)
Gram-positive 62 (11) 221 3 (2-4)
Beta-lactams 120 (21) 346 2 (2-3)
![Page 20: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/20.jpg)
Protocol WritingAimsSettingWhereWhoWhen
ProceduresWhat Allocation If… Then…
![Page 21: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/21.jpg)
Assessment (and dissemination) planOutcome selection Primary vs. secondary What matters to those involved in the process?
Primary comparison After you do it, what does it mean? Confounding, selection, measurement biases et al.
DesignIRB, consideration of benefits/risksData ManagementStatistical plan Sample size
![Page 22: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/22.jpg)
CONSORTConsolidated Standards of Reporting Trials
http://www.consort-statement.org/
![Page 23: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/23.jpg)
TRENDTransparent Reporting of Evaluations with Non-randomized Designs
American Journal of Public Health 94, no. 3 (March 1, 2004): pp. 361-366.
Infect Control Hosp Epidemiol 2016;37:1135–1140
![Page 24: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/24.jpg)
Is the procedure working?Process measurement Regular process data feedback to those involvedMake a goal and incentivize
Pilot and perfect: Estimates of sample size Dry runs Staggered roll out Collaboration during the study
![Page 25: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/25.jpg)
Nosce te ipsum: What are the limitations?How was randomization accomplished and was it effective? Contamination of study arms Selection or confounding bias
Outcome definition/ascertainment Loss to follow up
Adequately powered?How well was study protocol followed? Insufficient documentation/data collection Insufficient implementation data
Could study protocol be implemented in other practice settings? Generalizability Applicability External validity
![Page 26: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/26.jpg)
Special Considerations forinterventional antimicrobial stewardship research
Cannot blind Contamination and selection bias
Cannot deliver intervention without policy/system/practice change Unit of analysis Comparator
Don’t want to (or can’t) enroll/consent patientsSolution(s):Cluster trials: randomize to groups (units, practices, hospitals) instead of enrolling individuals (patients or providers) Stepped Wedge: random and sequential crossover of clusters from control to intervention until all clusters are exposed
![Page 27: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/27.jpg)
Special Considerations forinterventional antimicrobial stewardship research
Cannot blind Contamination and selection bias
Cannot deliver intervention without policy/system/practice change Unit of analysis Comparator
Don’t want to (or can’t) enroll/consent patientsSolution(s):Cluster trials: randomize to groups (units, practices, hospitals) instead of enrolling individuals (patients or providers) Stepped Wedge: random and sequential crossover of clusters from control to intervention until all clusters are exposed
![Page 28: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/28.jpg)
BMJ 2015;350:h391
![Page 29: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/29.jpg)
Special Considerations forinterventional antimicrobial stewardship research
Lack of generalizability Single center study
Solution(s):State in limitations, realize that the findings apply best to populations similar to your study Is there something that makes your study population special?
Make Table 1 a practical oneDescribe baseline ASP implementation or “control” arm well
![Page 30: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/30.jpg)
Special Considerations forinterventional antimicrobial stewardship research
Lack of generalizability Single center study
Solution(s):State in limitations, realize that the findings apply best to populations similar to your study Is there something that makes your study population special?
Make Table 1 a practical oneDescribe baseline ASP implementation, rates, or “control” arm well
![Page 31: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/31.jpg)
Special Considerations forinterventional antimicrobial stewardship research
Bias/Confounding
Solution(s):Anticipate possible confoundersMeasure them: Stratified randomization of the intervention Demonstrate they are “even” (or not differential) during both arms Add an additional control?
How much of your intervention effect might be biased? Post-hoc analyses – subgroups or sensitivity analysis By Site By patient subgroups
![Page 32: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/32.jpg)
Special Considerations forinterventional antimicrobial stewardship research
Bias/Confounding
Solution(s):Anticipate possible confoundersMeasure them: Stratified randomization of the intervention Demonstrate they are “even” (or not differential) during both arms Add an additional control?
How much of your intervention effect might be biased? Post-hoc analyses – subgroups or sensitivity analysis By Site By patient or provider characteristic
![Page 33: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/33.jpg)
Special Considerations forinterventional antimicrobial stewardship research
Limitations of AU rates as a primary outcome
SeasonalitySolution(s): Use a whole year of data in baseline/intervention periods Compare to similar season in baseline period Adjust for month/season in regression or use ITS Plan cross overs wisely
DenominatorsSolution(s): Make sure to report LOS Deal with early deaths DOOR/RADAR?
DOT
Patient daysDays present
![Page 34: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/34.jpg)
Special Considerations forinterventional antimicrobial stewardship research
Limitations of AU rates as a primary outcome
SeasonalitySolution(s): Use a whole year of data in baseline/intervention periods Compare to similar season in baseline period Adjust for month/season in regression or ITS Plan cross overs wisely
DenominatorsSolution(s): Make sure to report LOS Deal with early death DOOR/RADAR?
DOT
Patient daysDays present
![Page 35: DEVELOPING AND EVALUATING YOUR INTERVENTION · Slide: Dr. Tamar Barlam, Boston Univ. GRADE and PICO Develop PICO questions to frame topics Population of interest Intervention or indicator](https://reader035.fdocuments.us/reader035/viewer/2022071510/612efad11ecc51586943271e/html5/thumbnails/35.jpg)
Final ThoughtsStudy designs of any type can have impact on AS, as long as they are done well.AS, as a field, requires study designs that address implementation as well as effect. Read and utilize SHEA White Papers on Research Methods.Think through study limitations during intervention development and delivery. Continue to question!