Criteria to assess quality of observational studies evaluating the incidence, prevalence, and risk...
-
Upload
harry-robinson -
Category
Documents
-
view
214 -
download
0
Transcript of Criteria to assess quality of observational studies evaluating the incidence, prevalence, and risk...
Criteria to assess quality of Criteria to assess quality of observational studies evaluating the observational studies evaluating the
incidence, prevalence, and risk incidence, prevalence, and risk factors of chronic diseasesfactors of chronic diseases
Minnesota EPCClinical Epidemiology Methods Centre, Ottawa Health Research Institute, Ottawa
Tufts University Medical Center, BostonRTI International – University of North Carolina, Chapel Hill
Blue Cross and Blue Shield Association, ChicagoCenters for Disease Control and Prevention, Atlanta
Southern California EPC; RAND Corporation, Santa MonicaGerman Institute for Quality and Efficiency in Health Care, Köln, Germany
Johns Hopkins University, Baltimore
Analytical tasks
III. Collaboration with participating experts
I. Systematic literature review
II. Development of two checklists
IV. Finalizing checklists
Therapeutic vs. non therapeutic studiesPICOTS Non therapeutic studies Non randomized studies
Population General population Patients, health care recipients
Exposure (interventions, independent variables)
Risk factors that are independent of health care (allocation irrelevant)
Treatments allocation by investigators or health care professionals
Comparators Levels of risk Another trtm, usual care, standard care, previous trtm
Outcomes Incidence/prevalence of chronic conditions
Morbidity, mortality, quality of life, adverse events after treatments
Time Length of exposure Length of trtm, followup off the trtm
Settings General population Health care settings
Comparability (selection bias)
Not relevant Flaw
Intention-to treat Not relevant Flaw
Systematic literature review
• Systematic reviews and meta-analyses of observational nontherapeutic studies published in English in core clinical journals (145)
• Published tools (scales or checklists) for quality assessment of observational studies (96)
• Annotated bibliography of the methodological literature to identify biases and strategies to reduce bias in observational studies
Development of Checklists• Separate evaluation of external and internal
validity• Exhaustive range of criteria• Discriminating reporting and methodological
quality• Predefined major and minor flaws • Study and hypotheses level analyses• Automated reports• Templates for the protocols of quality
evaluations with predefined research specific quality standards
• Manual, help files, instructions
Collaboration with participating experts
• Criteria evaluation: face and content validity
• Pilot reliability testing (10 randomly selected articles): overall, by topic, by article, by domains of external and internal validity, and by quality component
• Discriminant validity: testing hypotheses of detected differences in quality and reporting vs. methodological quality
Finalizing checklists
• Checklist for studies of incidence or prevalence of chronic diseases (MORE)
• Checklist for studies of risk factors of chronic diseases (MEVORECH)
• Synthesis of evidence from the studies with major flaws:
• Level A exclusion from synthesis • Level B full review and quality abstraction • Level C separate limited synthesis if major flaws detected
Quality criteriaIncidence or prevalence studies
Aim of study Study designExternal Validity1. Sampling of the subjects by the
investigators2. Assessment of sampling bias 3. Estimation of sampling bias4. Exclusion rate from the analysis 5. Sampling bias is addressed in the analysis6. Subject flow Internal Validity1. Source to measure outcomes 2. Definition of outcomes3. Measurements of outcomes4. Outcomes in race, ethnic, age, or gender
subpopulations 5. Reporting of outcomes
Risk factors studiesAim of studyObjectivesStudy design External Validity1. Sampling of the subjects by the investigators2. Assessment of sampling bias 3. Estimation of sampling bias4. Exclusion rate from the analysis 5. Sampling bias is addressed in the analysis 6. Subject flow
Internal Validity1.Source to measure outcomes2. Definition of outcomes3. Measurements of outcomes4. Definition of the exposure 5.Measurements of the exposure 6. Confounding factors7. Loss of followup 8. Masking of exposure status9. Statistical analysis10. Appropriateness of statistical models11. Reporting of tested hypothesis12. Precision of the estimates13. Sample size justification
Major flawsExternal validity
Sampling in clinics forincidence in the generalpopulation;
Sampling of controls from differentpopulation as cases;
Response rate <40%;
Exclusion of >10% of eligible subjectsfrom the analyses
Reasons to exclude from the analysesdiffer for exposed and not exposed
Internal validitySeverity of the outcome can berelevant but not assessed in thestudy;
Non valid methods were obtained tomeasure the outcomes, exposure, orconfounding factors;
Different methods to measureexposure (risk factors, independentvariable) in cases and controls;
No strategies to reduce bias
Minor Flaws• External validity• Random sampling
restricted to geographic area
• Convenient sampling• The authors did not
assess sampling bias • Sampling bias was not
addressed in the analysis• Sampling of controls from
health care related sources
• Internal validity• Outcomes proxy reported• Obtained from medical
records (data mining) or administrative database
• Inter-methods validation • Subjective judgment of
reliability• Intensity/dose of exposure
can be relevant but not assessed in the study
Example of the reportArticle: _________________
Evaluator: _______________
External ValidityNot Reported
Addressing sampling bias Not reported
Sampling: General population based Not reported
Major flaw
Exclusion rate from the analysis >10%
Internal ValidityMajor flaw
Definition of incidence/prevalence: Severity Can be relevant but not assessed in the study
Minor
Definition of incidence/prevalence: Frequency of symptoms
Can be relevant but not assessed in the study
ConclusionsConclusions
• We propose two checklists for transparent and standardized quality assessment in systematic reviews and evidence-based reports
• Reasonable validity• Poor random overall and good
component reliability in a pilot testing of the articles from different topics
Future Research with Quality Assessment Using Proposed Checklists
• Protocols of systematic reviews of nontherapeutic observational studies with justified definitions of research specific quality standards and preplanned reliability testing
• Synthesis of evidence incorporating quality assessment (sensitivity of the results to quality)
• The evaluation of the level of evidence from several observational nontherapeutic studies
• Quality assessment of primary studies should improve quality of systematic reviews and evidence-based reports