Criteria for Assessing The Feasibility of RCTs. RCTs in Social Science: York September 2006...
-
Upload
naomi-bridges -
Category
Documents
-
view
214 -
download
0
Transcript of Criteria for Assessing The Feasibility of RCTs. RCTs in Social Science: York September 2006...
RCTs in Social Science: York September 2006
Today’s Headlines: “Drugs education is not working”
“ having reviewed research from across the world, a committee of doctors and scientists on the ACMD* concluded that the success of school-based schemes was "slight or non-existent" and could even be "counter-productive".
*Advisory Council on the Misuse of Drugs (ACMD)
RCTs in Social Science: York September 2006
Why are we here?•We share the goal of improving well being for society and individuals
• Interventions which we support as a society should benefit individuals and society and should not cause harm
•We value evidence
RCTs in Social Science: York September 2006
Who am I?•Economist•Managing Director of Matrix Research & Consultancy Ltd
•Member of the Campbell Crime and Justice Group
•Member of the Cochrane Campbell Economics Methods Group
•Advocate of improving the use of evidence to inform decisions
RCTs in Social Science: York September 2006
Who are you?•Politicians?
•Policy advisers?
•Civil Servants?
•Practitioners?
•Researchers?
•Economists…..?
RCTs in Social Science: York September 2006
Important evaluation questions (adapted: Bain 1999)
•Should it work? Theory
•Can it work? Implementation
•Does it work? Impact
• Is it worth it? Value
RCTs in Social Science: York September 2006
Source of interventions•Historical practice (“we have always done it this way”)
•Taught practice (“we teach people to do it this way”)
• Innovative practice (“I try new ideas”)
•Research (“I examine the theories”)
the popular vote………………
RCTs in Social Science: York September 2006
Role of trials for new medical interventions• Idea•Basic science•Laboratory trials•Clinical trials•Licence•Approval (e.g. NICE)
Average 10-15 years?
RCTs in Social Science: York September 2006
Particular challenges for trials in social science (adapted: Farrington 1983)
• Pace of idea to practice
• Theory base of treatment
• Definition and scope of treatment
• Context complexity
• Community vs individual interventions
• Hawthorne effects
• Contamination
• Outcome measures & measurement
• Duration and decay
RCTs in Social Science: York September 2006
Arguments for Trials• Interventions can do harm • If undertaken well trials minimize risk of bias
•Basic hypothesis: “can we identify an effect with confidence?”
• If results insignificant then:–There isn’t an effect; or–There is an effect but we haven’t detected it with confidence.
RCTs in Social Science: York September 2006
Arguments against trials•Epistemology: “the world doesn’t work like this”
•Analytical: “there are too many analytical constraints”
•Ethical: “you can’t deny treatment”
•Legal: “you might be challenged if you deny treatments”
•Logistical: “there are too many practical constraints
RCTs in Social Science: York September 2006
Research on FeasibilityResearch partners
•Matrix Research & Consultancy Ltd
•The Jerry Lee Centre, University of Pennsylvania.
Research base
•Collective research experience
•Home Office funded study for OBPs
RCTs in Social Science: York September 2006
Analytical challenges:Hierarchy
1. Internal validity: can we attribute effect to intervention?
2. Statistical power: can we measure the effect with confidence?
3. External validity: can we generalise the results?
RCTs in Social Science: York September 2006
Hurdles to ensure Internal Validity•Participant selection targeting
•Completion and attrition rates
• Inconsistent treatments & treatment measurement
•Multiple outcomes and inconsistent outcome measurement
• Independent, contamination-free alternative, treatments or no treatment
RCTs in Social Science: York September 2006
Hurdles for Statistical Power•Understanding expected/required effect size?And….Given this…•How to maximize statistical power given?
–Heterogeneity of sample?–Completion rates?–Attrition rates?
And…given this…•How many participants do you need?
RCTs in Social Science: York September 2006
Hurdles for External ValidityContext / Characteristics:
• Local service provider
•Participants
•Community
Tension between correcting for these and the increased challenges of achieving internal validity and statistical power.
RCTs in Social Science: York September 2006
Ethical Hurdles
• “denial of treatment is unethical”
• “validity of informed consent”
RCTs in Social Science: York September 2006
Legal Hurdles•E.g. for offenders tension between random assignment and requirements to sentence
•Scope of legal challenge:
–from those denied the intervention
–where programmes exist and are perceived to reduce risk to public/ stakeholders
RCTs in Social Science: York September 2006
Logistical Hurdles•Quality of trial staff
•Cost of a trial relative to the cost of the intervention and the value of the expected effect
•Sample size required when completion rates are low and attrition post completion is high
•Avoidance of Hawthorne effect
RCTs in Social Science: York September 2006
Are there Solutions?• Focus on particular well defined interventions• Appoint independent trained research teams • Randomise early• Separate control groups • Pick control treatments which enable effects to be measured• Manage and monitor implementation• Ensure programme stability • Determine sample size from desired effect size – look at the theory
and related research • Measure outcomes consistently • Identify cost effective study design• Educate and inform participants and stakeholders
RCTs in Social Science: York September 2006
If these aren’t available?Only then ask yourself:
“What is the next best alternative….and how can I minimize the risk of getting it wrong”
RCTs in Social Science: York September 2006
Final reflections•Researchers should recognise:
–decisions need to be made in real time–all information can provide evidence
•Users should recognise:–the risks of making the wrong decision increases as the quality of the evidence base declines
• In social science we are addressing the needs of people and communities who are vulnerable