A Systematic Review on Interventions Supporting Preceptor ...
Medicine, Nursing and Health Sciences Methods for synthesizing evidence of the effects of healthcare...
-
Upload
brianna-mason -
Category
Documents
-
view
220 -
download
3
Transcript of Medicine, Nursing and Health Sciences Methods for synthesizing evidence of the effects of healthcare...
Medicine, Nursing and Health Sciences
Methods for synthesizing evidence of the effects of healthcare interventions in systematic reviews of complex interventions (including statistical approaches)Joanne McKenzie, School of Public Health and Preventive Medicine19th Cochrane ColloquiumMadrid 2011
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 2
Google’s take on complexity …
www.nicholsoncartoons.com.au
Acknowledgements
Sue Brennan, Monash University, Australia
Sophie Hill, La Trobe University, Australia
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 4
Outline
Components of complexity in a systematic review
Summary to synthesis (pros and cons)
– Available methods
– Outcome categorisation
Conclusions and questions raised
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 5
Splitting Lumping
ConditionOne condition (e.g. diabetes)
Any condition
One setting (e.g. primary care)
Setting
All settings
Narrow(e.g. audit cycles)
Intervention
Any form(e.g. any QI
intervention)
One design(e.g. RCT)
Study design
Multiple designs
(e.g. RCT, ITS, CBA)
Consistent outcomes (e.g. test ordering)
Diverse outcomes
Outcome
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 6
Summary
Vote counting Summary of effect estimates
Meta-analysis
- texttabular
Synthesis
- texttabularharvest plots
- descriptive statisticsbox and whisker plots
- meta-analysispredictive intervalsforest plots
- sub-group analysismeta-regressiongraphical approaches
Exploring heterogeneity
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 7
Summary
Results of studies summarised:
• in the text of a publication without the use of a synthesis method
• in tables, providing a structured method for presenting data
Pros
Text: Provides an assembly of the available research meeting the inclusion/exclusion criteria.
Tables: More likely to report all results of all outcomes (i.e. may be less likely to selectively include results).
Results available for others to synthesize.
Cons
Text: Results summarised, not synthesized.
Little structure to reporting results may lead to selective reporting (privileging of findings above others).
Interpretation of results difficult/not possible.
Tables: Results summarised, not synthesized.
Overwhelming amount of information which is difficult for a reader to interpret (often multiple outcomes per study).
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 8
Summary
Results of studies summarised:
• in the text of a publication without the use of a synthesis method
• in tables, providing a structured method for presenting data
Pros
Text: Provides an assembly of the available research meeting the inclusion/exclusion criteria.
Tables: More likely to report all results of all outcomes (i.e. may be less likely to selectively include results).
Results available for others to synthesize.
Cons
Text: Results summarised, not synthesized.
Little structure to reporting results may lead to selective reporting (privileging of findings above others).
Interpretation of results difficult/not possible.
Tables: Results summarised, not synthesized.
Overwhelming amount of information which is difficult for a reader to interpret (often multiple outcomes per study).
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 9
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 10
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 11
Splitting Lumping
ConditionAny condition/aspect of care: preventive care, diabetes, asthma, hypertension, HIV, renal failure, palliative care, coronary heart failure, stroke, pain management, falls prevention, neonatal infection, anticipatory guidance, compensation, wait times
Setting
Any setting: ambulatory (22), inpatient/nursing home (10), mixed clinical (3), educational (4)
InterventionAny form: QI education for trainees (10), QI education for nontrainees (5), ‘other’ interventions with an educational component (5), education within a QI collaborative (19)
Study design
Multiple designs: RCT (8), nonrandomised trial (14), pre/post or time series (17)
Any outcome: attitude (6), knowledge (10), skill/behaviour (6), process (27), patient (18)
Outcome
Quality improvement education for clinicians
Boonyasai, JAMA 2007
[Boonyasai JAMA 2007]
Study 1
Study 2
Study 3
• 27/39 studies measured process outcomes (3 example studies)
• 15 clinical areas
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 13
Outcome categorisation (EPOC)
Example CQI review:
Healthcare professional performance (binary, continuous)
– e.g. adherence to recommended practice
Patient outcomes (binary, continuous)
– e.g. pain, quality of life, function, mortality
– e.g. patient experience of care, patient evaluation of care co-ordination, length of stay
Other outcomes
– e.g. resource use
[Brennan Cochrane Database Syst Rev 2009]
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 14
[Hill Wiley-Blackwell 2011]
Table 4.2: Outcomes of importance to consumers, communication and participation: a new taxonomy
Groups and orientation
Outcome category
(a) Consumer Knowledge and understandingCommunicationInvolvement in care Evaluation of careSupportSkills acquisitionHealth status and well beingHealth behaviourTreatment outcomes
(b) Healthcare provider
Knowledge and understandingConsultation processes
(c) Health service delivery
Service delivery levelRelated to researchSocietal or governmental
Outcome categorisation (CCRG)
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 15
SynthesisVote counting:• “Is there any evidence of an effect?”
• # studies showing harm compared with # studies showing benefit (regardless of stat. sig. or size of results)
• Sign test used to assess the stat. sig. of evidence of an effect in either direction
Pros
Provides a method for synthesizing effects when standard meta-analytical methods difficult to apply (e.g. variances of effect estimates not available).
Cons
Provides no information on the magnitude of effects (e.g. equal importance given to risk difference of 5% and 50%) .
No account of differential weighting across the studies.
Problems when stat. sig. used to define # positive and # negative studies (unit of analysis errors, underpowered studies).
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 16
SynthesisVote counting:• “Is there any evidence of an effect?”
• # studies showing harm compared with # studies showing benefit (regardless of stat. sig. or size of results)
• Sign test used to assess the stat. sig. of evidence of an effect in either direction
Pros
Provides a method for synthesizing effects when standard meta-analytical methods difficult to apply (e.g. variances of effect estimates not available).
Cons
Provides no information on the magnitude of effects (e.g. equal importance given to risk difference of 5% and 50%) .
No account of differential weighting across the studies.
Problems when stat. sig. used to define # positive and # negative studies (unit of analysis errors, underpowered studies).
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 17
[Boonyasai JAMA 2007]
Effectiveness of teaching quality improvement to clinicians
Harvest plots
Crowther Res Syn Meth 2011
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 19
SynthesisSummary of effect estimates:• “What is the range and distribution of effects?”
• ‘Median-of-medians’ approach (EPOC).
• One outcome chosen per outcome category (selection process independent of result & stat. sig.). Effect size associated with this outcome used to ‘characterise’ the outcome of the study.
Pros
Provides a method for synthesizing results when difficult to undertake a meta-analysis (e.g. missing variances of effects, unit of analysis errors).
Provides information on the magnitude and range of effects (IQR, range).
Cons
Does not weight effects; small studies are as influential as large studies.
Doesn’t use all available data for a particular outcome category.
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 20
SynthesisSummary of effect estimates:• “What is the range and distribution of effects?”
• ‘Median-of-medians’ approach (EPOC).
• One outcome chosen per outcome category (selection process independent of result & stat. sig.). Effect size associated with this outcome used to ‘characterise’ the outcome of the study.
Pros
Provides a method for synthesizing results when difficult to undertake a meta-analysis (e.g. missing variances of effects, unit of analysis errors).
Provides information on the magnitude and range of effects (IQR, range).
Cons
Does not weight effects; small studies are as influential as large studies.
Doesn’t use all available data for a particular outcome category.
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 21
[Farmer Cochrane Database Syst Rev 2008]
Printed educational materials: effects on professional practice and health care outcomes
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 22
[Jamtvedt Cochrane Database Syst Rev 2006 ]
Audit and feedback: effects on professional practice and health care outcomes
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 23
SynthesisMeta-analysis:• “What is the average intervention effect?” (random effects meta-analysis)
• Prediction intervals can be calculated to complement information of random effects meta-analysis; “What is the potential effect of an intervention in an individual study?”
Pros
Provides a combined estimate of average intervention effect (random effects), and certainty in this estimate (95% CI).
Weights estimates of effect; small studies are (generally) less influential compared with large studies.
Predictive intervals can be calculated; helpful when there is unexplained heterogeneity.
Forest plots display study effect estimates and CIs; can display pooled effect; familiar.
Cons
Requires variances of the effects.
Argued that a meta-analytic estimate (average effect) may be of little value when there is heterogeneity. Particularly if there is inconsistency in the direction of effect.
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 24
SynthesisMeta-analysis:• “What is the average intervention effect?” (random effects meta-analysis)
• Prediction intervals can be calculated to complement information of random effects meta-analysis; “What is the potential effect of an intervention in an individual study?”
Pros
Provides a combined estimate of average intervention effect (random effects), and certainty in this estimate (95% CI).
Weights estimates of effect; small studies are (generally) less influential compared with large studies.
Predictive intervals can be calculated; helpful when there is unexplained heterogeneity.
Forest plots display study effect estimates and CIs; can display pooled effect; familiar.
Cons
Requires variances of the effects.
Argued that a meta-analytic estimate (average effect) may be of little value when there is heterogeneity. Particularly if there is inconsistency in the direction of effect.
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 25
Exploring heterogeneitySub-group analysis, meta-regression, other statistical approaches:• “What factors modify the size of the intervention effect?”
• Can be used to investigate components (‘active ingredients’) of multifaceted interventions which may modify effects.
Pros
Provides hypotheses regarding what (set) of factors might be necessary for the intervention to be effective.
Cons
Observational analysis; may suffer confounding bias; aggregation bias; overfitting and spurious claims of association.
Investigation of intervention components when there are many is difficult (e.g. assumptions of additivity, correlation between combinations of components).
Requires variances of effects, measurement of factors.
Technical issues with baseline compliance.
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 26
Exploring heterogeneitySub-group analysis, meta-regression, other statistical approaches:• “What factors modify the size of the intervention effect?”
• Can be used to investigate components (‘active ingredients’) of multifaceted interventions which may modify effects.
Pros
Provides hypotheses regarding what (set) of factors might be necessary for the intervention to be effective.
Cons
Observational analysis; may suffer confounding bias; aggregation bias; overfitting and spurious claims of association.
Investigation of intervention components when there are many is difficult (e.g. assumptions of additivity, correlation between combinations of components).
Requires variances of effects, measurement of factors.
Technical issues with baseline compliance.
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 27
[Jamtvedt Cochrane Database Syst Rev 2006]
Audit and feedback: effects on professional practice and health care outcomes
Re-analysis of audit & feedback review
[Gardner Soc Sci Med 2010]
Authors used theory (behaviour change) to categorise intervention components (feedback, performance target, action plan), and investigated if components modified the effects.
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 29
Conclusions
Diversity of interventions, settings, conditions, outcomes, and study designs complicates the synthesis of evidence.
A range of ‘synthesis’ approaches are available; some are clearly better than others.
Limitations in quantitative synthesis should be acknowledged, but may be preferable to “qualitative interpretation of results, or hidden quasi-quantitative analysis …” [Ioannidis BMJ 2008]
Before making a decision not to synthesize data, review authors should consider what readers/decision makers might do (e.g. selection of favourable effects, count up #favourable results or stat. sig. results).
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 30
Conclusions
Diversity of interventions, settings, conditions, outcomes, and study designs complicates the synthesis of evidence.
A range of ‘synthesis’ approaches are available; some are clearly better than others.
Limitations in quantitative synthesis should be acknowledged, but may be preferable to “qualitative interpretation of results, or hidden quasi-quantitative analysis …” [Ioannidis BMJ 2008]
Before making a decision not to synthesize data, review authors should consider what readers/decision makers might do (e.g. selection of favourable effects, count up #favourable results or stat. sig. results).
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 31
Conclusions
Diversity of interventions, settings, conditions, outcomes, and study designs complicates the synthesis of evidence.
A range of ‘synthesis’ approaches are available; some are clearly better than others.
Limitations in quantitative synthesis should be acknowledged, but may be preferable to “qualitative interpretation of results, or hidden quasi-quantitative analysis …” [Ioannidis BMJ 2008]
Before making a decision not to synthesize data, review authors should consider what readers/decision makers might do (e.g. selection of favourable effects, count up #favourable results or stat. sig. results).
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 32
Conclusions
Diversity of interventions, settings, conditions, outcomes, and study designs complicates the synthesis of evidence.
A range of ‘synthesis’ approaches are available; some are clearly better than others.
Limitations in quantitative synthesis should be acknowledged, but may be preferable to “qualitative interpretation of results, or hidden quasi-quantitative analysis …” [Ioannidis BMJ 2008]
Before making a decision not to synthesize data, review authors should consider what readers/decision makers might do (e.g. selection of favourable effects, count up #favourable results or stat. sig. results).
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 33
Questions raised
Are arguments for not undertaking a meta-analysis based on too much clinical and methodological heterogeneity consistent with the ‘median-of-medians’ approach?
Are there other statistical approaches that may make better use of available data? E.g. meta-regression methods that adjust for correlated effects within studies (e.g. Hedges Res Syn Meth 2010).
The measures of effect used in complex reviews typically adjust for baseline imbalance (e.g. adj. RR, adj RD, adj OR). Do these estimators achieve the desired effect?
Will New Zealand win the 2011 Rugby World Cup?
October 2011Methods for synthesizing evidence of the effects of healthcare interventions 34
References
Boonyasai et al. Effectiveness of teaching quality improvement to clinicians: a systematic review. JAMA 2007;298(9):1023-37.
Brennan et al. Continuous quality improvement: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2009, Issue 4. 10.1002/14651858.CD003319.pub2.
Crowther M, Avenell A, MacLennan G, Mowatt G. A further use for the Harvest plot: a novel method for the presentation of data synthesis. Res Syn Meth 2011
Gardner B et al. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med 2010;70(10):1618-25.
Farmer et al Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2008(3): CD004398.
Hedges et al. Robust variance estimation in meta-regression with dependent effect size estimates. Res Syn Meth 2010;1(1):39-65.
Hill et al. Identifying outcomes of importance to consumers, communication and participation. In Hill S (ed). The Knowledgeable patient: Communication and participation in health. Wiley-Blackwell 2011.
Ioannidis JP et al. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ 2008;336(7658):1413-5.
Jamtvedt et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006(2):CD000259.