UOttawa.ca Deconstructing Rapid Reviews An Exploration of Knowledge, Traits and Attitudes Presented...
-
Upload
daniella-lloyd -
Category
Documents
-
view
219 -
download
0
Transcript of UOttawa.ca Deconstructing Rapid Reviews An Exploration of Knowledge, Traits and Attitudes Presented...
uOttawa.cauOttawa.ca
Deconstructing Rapid Reviews
• An Exploration of Knowledge, Traits and Attitudes
Presented by: Shannon Kelly, CADTH Symposium 2015 – Saskatoon, SK
uOttawa.ca
Faculty of MedicineSchool of Epidemiology, Public Health and Preventive Medicine
uOttawa.ca
Objectives
• To present results from 3 projects on rapid reviews conducted as part of MSc. Epidemiology at uOttawa
I. Defining rapid reviews – a consensus approach
II. Attitudes and perceptions towards rapid reviews
III. Defining rapid reviews – a pragmatic approach
2
uOttawa.ca
RAPID REVIEWS?Why do we care about
uOttawa.ca
What is a rapid review?
“…a streamlined approach to synthesizing evidence in a timely manner - typically for the purpose of informing emergent decisions faced by decision makers in health care settings.”
~ Khangura, 2012
Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Systematic Reviews. 2012;1:10.
uOttawa.ca
Rapid reviews influence decision-making in Canada
• Knowledge users in Canada (everywhere) need support to make timely, evidence-informed health care decisions
• Systematic reviews and HTA often take too long!
• Delay important policy decisions
• Use less-than-best evidence
• Rapid reviews help fill this void
• Balance rigour, relevance, timelines
…We still need to know so much more…
uOttawa.ca
What do we know?
Ultra-rapid reviews, first test results
uOttawa.ca
What don’t we know?
“While rapid review approaches are being used by HTA producers across the world in a variety of ways an agreed-upon definition describing its constitution and methods is lacking” (Khangura, 2014)“While the concept
of rapid evidence synthesis, or rapid review, is not novel, it remains a poorly understood and yet ill-defined set of disparate methodologies supported by a paucity of published, available scientific literature”(Khangura, 2014)
“..evidence is only slowly emerging as to which steps in the systematic review process may be altered by increased speed” (Schünemann, 2015)
“In order to consider offering rapid reviews, one first has to be able to define what they are. Previous reviews on the topic of rapid reviews describe them as ‘ill-defined,’ ‘not well-defined,’ lacking a single definition, or ‘varying widely in terms of the language used to describe them.’” (AHRQ, Hartling, 2015)
“…a literature search by the authors does not elucidate any clear or final definition of what a RR is and how exactly the methodology of a RR differs from a full SR.” (Harker, 2012)
“There is little empirical evidence comparing the continuum of products among rapid reviews and full systematic reviews, or analysing the diverse methods used in rapid reviews. It is important not only to establish transparent methodologies for rapid reviews, but also to understand the implications of what is lost in terms of rigour, bias, and results when methods associated with full systematic review are streamlined.” (Ganaan, 2010)
uOttawa.ca
Needed a research program to improve knowledge on rapid reviews
An improved understanding of rapid reviews will help us better meet the needs of decision-makers
uOttawa.ca
DEFINING RAPID REVIEWSA consensus approach
uOttawa.ca
Objective
• The study will employ a modified Delphi method to survey of experts in the field of evidence synthesis and rapid reviews to discover what they consider the key defining characteristics of rapid reviews to be.
Note: For our purposes, defining characteristics are:
‘one of a number of essential features by which a rapid reviews can be recognized’.
uOttawa.ca
Method: The Delphi Technique
Definition: a procedure that structures a communication process among a group of experts.(Linstone & Turoff, 1975)
Characteristics of the Delphi technique: Staged process, assessment of agreement or consensus Assessment of the group’s view; An opportunity for informants to revise their views; Opportunity to react to and assess differing view points; Anonymity of informants.
Panelists are selected based on their expertise in the subject matter; not random.
uOttawa.ca
Participants
Experts targeted for participation were:
1. Subject-matter experts or experienced researchers with knowledge of a variety of evidence synthesis methods and practical experience with rapid reviews;
2. Authors with publications relevant to rapid reviews; and,
3. Delegates presenting pertinent work at recent meetings, workshops, conferences or symposia.
• International Cochrane Colloquium, Health Technology Assessment international (HTAi), CADTH Symposium, Cochrane Canada Symposium).
uOttawa.ca
The Modified Delphi Process
STUDY SCOPING AND EXPERT PANEL DEFINITION
ROUND 1: WEB-BASED SURVEY ON KEY
RR THEMES
(FluidSurveys)
ROUND 2:DRAFT CHARACTERISTIC
STATEMENTS
(email)
ROUND 3:REVISED STATEMENTS, EXPERT REVISION OF
AGREEMENT
(email)
FINAL 11 CHARACTERISTICSTATEMENTS
PUBLICATIONPENDING
Review agreement, refine statements
120 POTENTIAL EXPERTS
Pilot Review, summarize,
analyze, draft statements66 EXPERTS 44 EXPERTS
26 EXPERTSCONSENSUS ON 10 OF 11 STATEMENTS
Consensus level = 70%, Approved by TOHREB
Review agreeme
nt
Summarize results
uOttawa.ca
Results: Key Characteristics (1)
• Conducted in less time than a systematic review;
• Have a protocol (at least describing objectives, scope, PICO, approach);
• Tailor the explicit, reproducible methods conventionally used in a systematic review in some manner to expedite the process to meet the needs of a decision-maker;
• Use a spectrum of approaches to complete an evidence synthesis related to a defined research question(s) using the most systematic or rigourous methods the limited time frame will allow.
Rapid reviews:
uOttawa.ca
RR time to completion
Delphi Expert Opinion
Previous RR Samples GANAAN
2010HARKER
2012INAHTA scan
2012POLISENA
2015AHRQ2015
TRICCO (RR SYMP.
CADTH)
Minimum 3 wk >3 mo 1 wk 1 wk 5 min <1mo
Maximum 6 mo < 24 mo 12 mo 12 mo 8 mo 12 mo
Rapid Review
Systematic Review
0 10 20 30 40 50 60
Time (in weeks)
Khangura et al. (2012): SRs take 6 months to 2 years, and RRs ≤ 5 weeks.
Delphi panel: Median time for SR is 36 weeks, while median for RR is 9 weeks.
Min = minute; d = days, wk = week; mo = month; yr = year
uOttawa.ca
Results: Key Characteristics (2)
• Balance the risk of the yet undetermined impact of workflow adaptations to the validity of the review and its results, with the needs of the knowledge user;
• Report methods and findings transparently, with appropriate detail required to
– answer the research question;– meet the requirements of the knowledge user and
allow assessment of confidence in the results; – inform the audience for which the review is intended;– while meeting delivery time line agreed upon in
advance.Consensus (72 - 100%)
uOttawa.ca
Still up for debate…• Tension: risk of bias/critical appraisal;
– individual study or review level
– distinct and opposing camps
• Achieved consensus on no consensus – RRs as a unique method;
• Time is still relative…;• Protocol format, publication/registration;• Peer review;• Unpublished (grey) literature;• Not all rapid reviews meet this definition? Overlap with
some key features of SR, HTA.
Statements based only on the experience of the expert panel…must be balanced with published literature.
uOttawa.ca
RAPID REVIEWSExploring attitudes and perceptions towards
uOttawa.ca
Method: Q methodology
A research technique that permits the systematic study of subjectivity (opinions, beliefs and preferences).
INTERPRETATION
FACTOR ANALYSIS
RANK ORDER OF THE STATEMENTS: Q SORT
DEVELOPMENT OF REPRESENTATIVE STATEMENTS: Q SET
DEFINING STUDY PARTICIPANTS: P SET
COLLECTION OF OPINION STATEMENTS: CONCOURSE
RESEARCH QUESTION
*Adapted from Amin, 2000
uOttawa.ca
Conceptual Q sort layout
We replicated this process using online Q methodology tools:
uOttawa.ca
• When time allows, a comprehensive systematic review of all available evidence should always be conducted.
• Further research comparing the methods and results of rapid reviews and systematic reviews is required before I decide how I feel about rapid reviews.
• Rapid reviews mean different things to different people.
• Rapid reviews should only precede a more comprehensive and rigorous systematic review.
• The opportunity cost of a comprehensive SR or HTA is too high and it is more advantageous to conduct rapid reviews when timeliness is a factor.
• All evidence synthesis products, including rapid reviews, SRs, or HTAs, can be conducted very well or very poorly.
• Rapid reviews are comparable to SRs except they are done in a more timely fashion.
• Rapid reviews are ‘quick and dirty' systematic reviews.
• Rapid reviews need to be tailored to the specific needs of the knowledge user.
• Rapid reviews meet the needs of knowledge users.
• It is always appropriate to conduct a rapid review.
• It is important to have minimum standards for the reporting of rapid reviews (e.g., a PRISMA-RR).
• Standardization of rapid review methods may conflict with the needs of knowledge users
• A good quality review of evidence is determined by the methods used, not by the speed at which it is completed.
• It is difficult to tell a rapid review from a systematic review unless very specific nomenclature is used in the title or description of methods.
• A rapid review cannot be a systematic review.
Sample statements
uOttawa.ca
Summary
• 11 participants performed the online Q sort
• 50 statements representinga range of viewpoints andand subjective opinion on RRs
• Following factor analysis3 factors extracted
Geographic Location n (%)
Canada 11 (100)
Age: n (%)
18 to 35 2 (18.2)
36 to 50 4 (36.4)
50 or above 5 (45.5)
Sex: n (%)
Female 8 (72.7)
Education: n (%)
Doctorate 7 (63.6)
Masters 3 (27.3)
Undergraduate 1 (9.1)
Consider Themselves: n (%)
Researcher/Producer 8 (72.7)
Knowledge User 2 (18.2)
Neither 1 (9.1)
Have ever been the author of a rapid review
n (%)
Yes 7 (63.6)
No 4 (36.4)
Unsure 0 (0)
Have used a rapid review to aid in a policy or decision-making?
n (%)
Yes 7 (63.6)
No 3 (27.3)
Unsure 1 (0.9)
uOttawa.ca
Title
uOttawa.ca
Title
uOttawa.ca
Factor 3. “Pragmatism over principle”
uOttawa.ca
Consensus Statements
Agree6. Rapid reviews mean different things to different people.
Disagree7. Rapid reviews should only precede a more
comprehensive and rigorous systematic review.34. It is appropriate to endeavor to define a single, unique
methodology for rapid reviews.42. Any review of evidence that takes longer than 3 months
to produce is not a rapid review.43. Any review of evidence that takes longer than 1 month to
produce is not a rapid review.
uOttawa.ca
DEFINING RAPID REVIEWSA pragmatic approach
uOttawa.ca
Summary of results
• Review of RRs published in journals or as reports by various agencies and organizations worldwide;
– Supplemented with INAHTA scan to query HTA agencies.
• Screened 1,800+ database records and grey literature results - Single reviewer, checking by a second;
• Included RR samples were examined for completion time, nomenclature, study characteristics, approach/tailoring and other relevant features, including whether the results are contextualized or framed with policy implications.
uOttawa.ca
In general
• Findings matched previous work on RR methods:– “ One species, many breeds” Heterogeneity of approaches,
nomenclature – supports the evidence continuum concept;– May find more than one approach within a single
organization (e.g., CADTH);– Methods not well-reported; Self-identifying RRs with no way
to assess what was tailored.• In addition:
– RRs are getting published;– Need to focus on contextualization of results, policy
implications for end-user;– Starting to see RR protocols published (n=4) or registered
in PROSPERO (n=3).
uOttawa.ca
Summary• Addressed RR knowledge gaps with 3
separate but interrelated studies;
• Experts have identified defining characteristics that may help differentiate RRs from other evidence synthesis approaches;
• Agree to disagree in some areas;
• Three distinct viewpoints on RRs were ascertained during a Q methodology study of subjective opinion;
• A selection of RR samples show that there is room for improvement (without compromising on the needs of end-users).
uOttawa.ca
Questions? Thoughts?
A special thank you to Dr. Tammy Clifford and Dr. David Moher, to CADTH for the initial funding for this student project, and
everyone who has participated in surveys, piloted questionnaires, shared data and supported this research with their own time