Assessing Quality of Individual Studies
-
Upload
effective-health-care-program -
Category
Health & Medicine
-
view
520 -
download
10
Transcript of Assessing Quality of Individual Studies
Assessing the Quality of Individual Studies
Prepared for:
The Agency for Healthcare Research and Quality (AHRQ)
Training Modules for Systematic Reviews Methods Guide
www.ahrq.gov
Systematic Review Process Overview
To describe the concept of quality assessment
To identify reasons for quality assessment
To list the steps in quality assessment
To describe and report the methods for quality assessment
Learning Objectives
Definition of quality:“[T]he extent to which all aspects of a study’s design and conduct
can be shown to protect against systematic bias, nonsystematic bias, and inferential error.” (Lohr & Carey, 1999)
Considered to be synonymous with internal validity
Relevant for individual studies
Distinct from assessment of risk of bias for a body of evidence
What Is Quality Assessment?
Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.
Systematic Errors: Include selection bias and confounding, in which values tend to be
inaccurate in a particular direction
Nonsystematic Errors:Are attributable to chance
Inferential Errors:Result from problems in data analysis and interpretation, such as
choice of the wrong statistical measure or wrongly rejecting the null hypothesis
What Are the Components of Quality Assessment?
Lohr KN, Cary TS. Jt Comm J Qual Improv 1999;25:470-9.
Consider the Contribution of an Individual Study to the Body of Evidence
Quality (systematic error and inferential error, type of study)
Size of study (nonsystematic or random error)
Direction and magnitude of results
Relevance of results to key questions
Risk of Bias
Precision
Directness
Consistency
Applicability
Owens DK, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2009_0805_grading.pdf.
Direct vs. indirect comparisons, health outcomes vs. surrogate outcomes
Quality assessment is required for:Interpreting results
Grading the strength of the body of evidence
Quality assessment may also be used for:Selecting studies for the review (based on a priori assessment of
evidence gaps)
Selecting studies for qualitative synthesis
Selecting studies for quantitative synthesis
Interpreting heterogeneous findings
Reasons for Quality Assessment
1. Classify the study design
2. Apply predefined criteria for quality assessment of each outcome based on:Study design — sources of bias may vary by design
Study conduct — poor study conduct and discrepancy between design and conduct may increase risk of bias
Reporting — quality assessment may be influenced by adequacy of reporting when information on study design and conduct are missing
What Are the Steps in Quality Assessmentof Each Individual Study (I)?
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
.
3. Arrive at a summary judgment of the study’s quality to rate studies good, fair, or poorRequires resolution of conflicts when two reviewers
independently evaluate quality
What Are the Steps in Quality Assessment of Each Individual Study (II)?
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
Did the study have more than one group or arm? If so, was a control group present?
Did investigators have control over allocation and timing of the intervention?
Did investigators randomly allocate subjects to interventions?
Did investigators concurrently measure intervention and exposure status for intervention and comparison groups?
Did investigators concurrently measure outcomes for intervention and comparison groups?
Questions To ConsiderWhen Classifying Study Design
Select a tool that is based on coverage of important criteria
Based on the topic, select and apply one of several available tools that consider and explain how to evaluate:Similarity of groups at baseline in terms of baseline characteristics
and prognostic factors
Validity of primary outcomes
Blinded measurement of outcomes
Apply Predefined Criteria for All Study Types (I)
Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
Apply one of several available tools that considerIntention-to-treat analysis
Differential loss to followup between the compared groups or overall high loss to followup
Conflict of interest
Apply Predefined Criteria for All Study Types (II)
Methods used for randomization
Allocation concealment
Blinding of subjects and providers
Additional Criteria for Trials
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
Sample size, width of confidence intervals, or power
Methods for selecting participantsInception cohort, methods to adjust for or avoid selection bias
Methods for measuring exposure variables
Additional Criteria for Observational Studies (I)
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
Methods for dealing with any design-specific issues such as recall bias and interviewer bias
Analytical methods to control confoundingMatching, stratification, multivariate analysis, or other statistical
adjustment
Additional Criteria for Observational Studies (II)
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
After assessment of individual criteria, assign ratings of “good,” “fair,” or “poor” (attributes described in later slides)
Assess quality for each outcome of interest
Base ratings on the evaluation of likely effect of design or execution flaws on internal validity, rather than a nominal failure to meet every quality criterion
Adjudicate differences between raters in a transparent manner when two raters independently assess overall quality
Arrive at a Comprehensive Judgment of Quality
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
Design and conduct of study address risk of bias
Appropriate measurement of outcomes
Appropriate statistical and analytical methods
Low drop-out rates
Adequate reporting of statistical and analytical methods, drop-out rates and reasons, and outcomes (no reporting errors)
Attributes of Good-Quality Studies
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
Do not meet all the criteria required for a rating of good quality
No flaw is likely to cause major bias
Missing information often drives rating
Attributes of Fair-Quality Studies
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
Significant biasesInappropriate design, conduct, analysis, or reporting
Large amounts of missing information
Discrepancies in reporting
Attributes of Poor-Quality Studies
Helfand M, Balshem H. In: Helfand M, Balshem H. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf.
Poor-quality studies may be excluded or included
Base decisions on gaps in current evidence and availability of good-quality or fair-quality studies
Justify selective inclusion of poor-quality studies for subgroups or subquestions
Treatment of Poor-Quality Studies in the Review
Accompany overall quality rating for individual studies with a statement of:Flaws in the design or execution of a study
The potential consequences of that flaw
Report the criteria and the process used to arrive at a quality rating
Reporting Quality Ratings
Quality assessment:Is synonymous with internal validity
Refers to individual studies
Contributes to, but is separate from, the evaluation of the risk of systematic bias for the body of evidence
Key Messages (I):Definition of Quality Assessment
Results of quality assessment are used in multiple steps in the systematic review process, from final inclusion of studies to interpretation of evidence
Steps in quality assessmentStudy design classification
Assessment of individual quality criteria
Summary judgment of the study quality
Key Messages (II): Rationale of and Steps in Quality Assessment
Transparency of processFull reporting of all elements of quality for each individual study
Explicit description (and examples) of how each criterion was operationalized
Clear reporting of how team members scored quality
Description of how conflicts between raters were resolved
Transparency of judgmentExplanation of final rating
Key Messages (III): Reporting
Deeks JJ, Dinnes J, D’Amico R, et al, for the International Stroke Trial Collaborative Group and the European Carotid Surgery Trial Collaborative Group. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7(27):iii-x,1-173.
Hartling L, Bond K, Harvey K, et al. Developing and testing a tool for the classification of study designs in systematic reviews of interventions and exposures. (Prepared by the University of Alberta Evidence-based Practice Center under contract no. 290-02-0023.) In press.
Helfand M, Balshem H. Principles in developing and applying guidance. In: Methods guide for comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality, Posted August 2009. Available at: http://www.effectivehealthcare.ahrq. gov/ehc/products/ 60/294/2009_0805_principles1.pdf.
References (I)
Lohr KN, Carey TS. Assessing “best evidence”: issues in grading the quality of studies for systematic reviews. Jt Comm J Qual Improv 1999;25:470-9.
Owens DK, Lohr KN, Atkins D, et al. Grading the strength of a body of evidence when comparing medical interventions. In: Methods guide for comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality, Posted July 2009. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2009_0805_grading.pdf.
Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007;36:666-76.
References (II)
West S, King V, Carey TS, et al. Systems to Rate the Strength of Scientific Evidence, Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute–University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011. Rockville, MD: Agency for Healthcare Research and Quality, March 2002. AHRQ Publication No. 02-E015. Available at: http://www.ahrq.gov/clinic/epcsums/strengthsum.pdf.
Whiting P, Rutjes AWS, Dinnes J, et al. Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess 2004;8(25):iii, 1-234.
References (III)
This presentation was prepared by Meera Viswanathan, Ph.D., a member of the Research Triangle Institute–University of North Carolina Evidence-based Practice Center.
The presentation is based on an update of chapter 6 in version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (update available at: http://www.effectivehealthcare. ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf).
Author