Post on 12-Apr-2017
The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org
Howard WhiteCEO, Campbell Collaboration
@washeval @c2update @HowardNWhite
The importance of systematic reviews
The Campbell Collaboration www.campbellcollaboration.org
The Campbell Collaboration www.campbellcollaboration.org
The seven piece study
But these are observational data, which don’t control for selection bias (people who eat more than five portions a day are wealthy, educated, health fanatics)
The five piece study
This is a systematic review, using data from 16 high-quality studies (observational data but analysis controls for confounders)
The Campbell Collaboration www.campbellcollaboration.org
The Campbell Collaboration www.campbellcollaboration.org
The Campbell Collaboration www.campbellcollaboration.org
WHAT IS GOING ON HERE?
The Campbell Collaboration www.campbellcollaboration.org
Errors in hypothesis testingH0 correct H0 false
Don’t reject H0 No error Type II error‘false negative’
Reject H0 Type I error‘false positive’
No error
The Campbell Collaboration www.campbellcollaboration.org
Hypothesis testing: if H0 true
H0: ß = 0
Don’t reject H0
5% chance that when H0 true get a sample leads you to reject H0
The Campbell Collaboration www.campbellcollaboration.org
Errors in hypothesis testing
H0 true H0 false
Don’t reject H0 No error Type II error
Reject H0 Type I error = 5%
No error
The Campbell Collaboration www.campbellcollaboration.org
Hypothesis testing HA correct
0Incorrectly reject HA approx 40% of the time
Power = 1 – type II error
HA
The Campbell Collaboration www.campbellcollaboration.org
Errors in hypothesis testing
H0 true H0 false
Don’t reject H0 No error Type II errorMaybe 20% but often
40-60%
Reject H0 Type I error = 5%
No error
The Campbell Collaboration www.campbellcollaboration.org
The horrifying truth about hypothesis testing
• If the ‘null hypothesis’ is correct (null = no programme impact) then we will correctly agree with the null 95% of the time (we are wrong 5% of the time)
• But if the null hypothesis is wrong (the programme works) then we probably incorrectly conclude the programme doesn’t work 40-60% of the time!!!
The Campbell Collaboration www.campbellcollaboration.org
Implications
• An under-powered RCT is no better than tossing a coin at determining if a successful programme is working so
• Power, power, power• We also need replicate ‘unsuccessful’ programmes• And we really REALLY need to do systematic reviews with
meta-analysis….
The Campbell Collaboration www.campbellcollaboration.org
Pooling evidence
1 = no effect
Intervention worksIntervention is harmful
So pooling data allows us to overcome the high risk of Type II error
And goal scoring can be very misleading (DON’T do it)
The Campbell Collaboration www.campbellcollaboration.org
A real life example
30-50% reduction in mortality
Corticosteriod for women about to deliver prematurely
The Campbell Collaboration www.campbellcollaboration.org
So let’s look at the cancer graph again …
The Campbell Collaboration www.campbellcollaboration.org
Eggs
The Campbell Collaboration www.campbellcollaboration.org
So not much gives you cancer…
The Campbell Collaboration www.campbellcollaboration.org
Source: Li-Xuan Sang et al. Consumption of coffee associated with reduced risk of liver cancer: a meta-analysis BMC Gastroenterol. 2013; 13: 34 doi: 10.1186/1471-230X-13-34
COFFEE AND LIVER CANCERBut coffee is good for you
The Campbell Collaboration www.campbellcollaboration.org
And this really matters… growing number of studies show most things don’t work.
• Education: 90 interventions evaluated in RCTs by IES - 90% had weak or no positive effects.
• Employment/training: Department of Labor-commissioned RCTs 75% weak or no positive effects
• Business: Over 13,000 RCTs of new products/strategies conducted by Google and Microsoft, 80- 90% no significant effects.
But are these sufficiently powered???Need to combine the evidence
The Campbell Collaboration www.campbellcollaboration.org
There’s more to a systematic review than meta-analysis
- Systematic search- Systematic screening- Systematic coding- Systematic synthesis- Systematic presentation of resultsNot being systematic introduces bias – as we shall see
The Campbell Collaboration www.campbellcollaboration.org
What the evidence synthesis process should look like
Source: Julia Littell: Campbell Systematic Reviews: Evidence for Implementation and Impact, GIC Dublin 2015
The Campbell Collaboration www.campbellcollaboration.org
What the evidence synthesis process actually looks like
Source: Julia Littell: Campbell Systematic Reviews: Evidence for Implementation and Impact, GIC Dublin 2015
Over-emphasis on significant findings
Studies with significant findings are 2-3 times more likely to be published
Selective presentation of results
The Campbell Collaboration www.campbellcollaboration.org
An example: the treatment of results from a single study of parent training (PT) versus multi-systemic training (MST) (a branded programme)
RCT assigning 43 abusive or neglectful families to either:• Parent training: group sessions discussing parenting
techniques• Multi-systemic therapy: individual family treatment tackling
multiple issues, e.g expectation re. child behaviour, child management, emotional support, parental behaviour change
Study looked at 30 outcomes on individual and child functioning, stress etc.
The Campbell Collaboration www.campbellcollaboration.org
Outcome reporting and confirmation bias in action
9 out of 14 reviews report just one outcome from the paper, favouring MST
Source: Julia Littell: Campbell Systematic Reviews: Evidence for Implementation and Impact, GIC Dublin 2015
The Campbell Collaboration www.campbellcollaboration.org
And how reviewers summarized the Brunk et al. paper…
Source: Julia Littell: Campbell Systematic Reviews: Evidence for Implementation and Impact, GIC Dublin 2015
No difference
The Campbell Collaboration www.campbellcollaboration.org
What the systematic review says
Out of home placement: no differenceDelinquency: no difference
Family cohesion: no difference
The Campbell Collaboration www.campbellcollaboration.org
Systematic reviews rebalance the evidence pyramid
Narrativereviews
More than 100 narrative reviews most stating MST is more effect than alternatives
MST is not consistently better or worse than alternatives
The Campbell Collaboration www.campbellcollaboration.org
So drop narrative reviews in favour of systematic reviews to rebalance the evidence pyramid
More than 100 narrative reviews most stating MST is more effect than alternatives
MST is not consistently better or worse than alternatives
The Campbell Collaboration www.campbellcollaboration.org
And in some areas, SRs have already made a difference
Crime and justice
1970s “Nothing works”Analysis of 231 studies:“With few exceptions, the rehabilitative efforts that have been reported so far have no appreciable effect on recidivism”Lipton, Martinson and Wilks ‘The Effectiveness of Correctional Treatment, 1975
Abandonment of rehabilitation in US and other countries (in prison at least get an incarceration effect)
But ….
The Campbell Collaboration www.campbellcollaboration.org
The analysis was goal scoring:
and there is no mixed evidence, only poorly synthesized evidence. Meta-analysis gave clear results
though in fact 48% of studies found significant positive effects
showed ‘mixed evidence’ for all categories of intervention
The Campbell Collaboration www.campbellcollaboration.org
Rehabilitation works
Review of 9 meta-analyses found ALL had positive average treatment effect
Source: Lipsey and Wilson (1993)
And prison no better than non- custodial sentences… or possibly worse
RCTs & 1 natural experiment
PSM
Source: Villletaz et al., 2013
The Campbell Collaboration www.campbellcollaboration.org
Which means that…
• Prison is at best no better and possibly worse than non-custodial sentences
• Each additional year of prison increases recidivism by 3-4 per cent
Source: Petrosino et al. Scared Straight and Other Juvenile Awareness Programs for Preventing Juvenile Delinquency: A Systematic Review Campbell Systematic Reviews 2013:5
‘Shock approaches’ such as boot camps and scared straight are unsuccessful and even harmful
The Campbell Collaboration www.campbellcollaboration.org
And this evidence is being used
“We must use sound, research-based, rehabilitation programmes for offenders so they do not re-offend.”
The Campbell Collaboration www.campbellcollaboration.org
Similar story in policing
1997 Sherman et al. Preventing Crime: What works, what doesn’t, reviewing over 500 crime prevention programmes
UK £250 million ‘Crime reduction programme’
Source: Braga et al. Hot spots policing effects on crime. Campbell Systematic Reviews 2012:8
For example hotspot policing
The Campbell Collaboration www.campbellcollaboration.org
Reviews can be used to answer both first generation questions (does it work) and second generation (design) questions.
Examples of looking at design questions
The Campbell Collaboration www.campbellcollaboration.org
First generation: payment for environmental services
Tiny effect
0.3 % reduction in deforestationi.e. after 10 years 97% of land for which payments received would still have been forested in absence of the programme
The Campbell Collaboration www.campbellcollaboration.org
What works by type of programme for teenage pregnancy
Source: Scher et al. Interventions Intended to Reduce Pregnancy-Related Outcomes Among Adolescents. Campbell Review 2006:12
The Campbell Collaboration www.campbellcollaboration.org
Impact of welfare for work schemes by administration
• Source: Smedslund et al Work Programs for Welfare recipients. Campbell Review 2006:9
Employment
Earnings
Welfare payments
Welfare proportion
0.00 0.20 0.40 0.60 0.80 1.00 1.20
Bush Clinton Reagan 2nd Reagan 1st Carter Ford Johnson
The Campbell Collaboration www.campbellcollaboration.org
Second generation: conditional cash transfers (CCTs)
• Mexico: Progressa launched 1996 (renamed Oportunidades)
• Brazil: 12 million families by 2010Cash payment on
conditions:• Education with 80%
attendance and maintaining certain grade
• Health: Ante-natal care, child immunization
Targeted both geographically and by means test
Design questions:• Do conditions
matter?• Timing, nature and
size of payment• Who to give it to
The Campbell Collaboration www.campbellcollaboration.org
Meta-analysis allows us to get at design features, for example …CCTs have a larger effect on enrolment rates• Secondary than primary• The larger the transfer• The less frequent the transfer• If conditions include achievement not just attendance
And…
The Campbell Collaboration www.campbellcollaboration.org
Conditionality works
Children 60% more likely to be in school with conditionality which is monitored and enforced compared to no conditions
But we need a lot of primary studies to exploit heterogeneity
The Campbell Collaboration www.campbellcollaboration.org
Water, supply and sanitation interventions4 main types of intervention:
– Water supply improvement: source or point-of-use– Water quality: water treatment/protection at source or
point-of-use (households)– Sanitation: provision of facilities (improved latrines,
sewer connection)– Hygiene: soap, hygiene education
Usual outcome variable is child diarrhoea
The Campbell Collaboration www.campbellcollaboration.org www.3ieimpact.orgAuthor name44
The Campbell Collaboration www.campbellcollaboration.org
45
Effectiveness results pooled (outcome = child diarrhoea)
NOTE: Weights are from random effects analysis
Water supply interventions
Subtotal
Water quality interventions
Subtotal
Sanitation interventions
Subtotal
Hygiene interventions
Subtotal
Multiple interventions
Subtotal
ID
Study
0.98 (0.89, 1.06)
0.58 (0.50, 0.67)
0.63 (0.43, 0.93)
0.69 (0.61, 0.77)
0.62 (0.46, 0.83)
ES (95% CI)
0.98 (0.89, 1.06)
0.58 (0.50, 0.67)
0.63 (0.43, 0.93)
0.69 (0.61, 0.77)
0.62 (0.46, 0.83)
ES (95% CI)
Ratio favours intervention 1.1 .5 .75 1 2
BUTEvidence largely from trials not actual projects
And hints of weak sustainability
The Campbell Collaboration www.campbellcollaboration.org
Sustainability 1: less impact over longer periods
NOTE: Weights are from random effects analysis
Water supply (12 months or more)
Subtotal
Water quality (under 12 months)
Subtotal
Water quality (12 months or more)
Subtotal
Sanitation (12 months or more)
Subtotal
Hygiene (under 12 months)
Subtotal
Hygiene (12 months or more)
Subtotal
Multiple (under 12 months)
Subtotal
Multiple (12 months or more)
Subtotal
ID
Study
0.82 (0.71, 0.96)
0.56 (0.47, 0.66)
0.81 (0.67, 0.97)
0.64 (0.37, 1.10)
0.72 (0.60, 0.86)
0.67 (0.49, 0.91)
0.41 (0.23, 0.74)
0.77 (0.70, 0.85)
ES (95% CI)
0.82 (0.71, 0.96)
0.56 (0.47, 0.66)
0.81 (0.67, 0.97)
0.64 (0.37, 1.10)
0.72 (0.60, 0.86)
0.67 (0.49, 0.91)
0.41 (0.23, 0.74)
0.77 (0.70, 0.85)
ES (95% CI)
Ratio favours intervention 1.1 .5 .75 1 2
The Campbell Collaboration www.campbellcollaboration.org
Sustainability 2: low compliance after a while
• Ceramic filter provision in Cambodia; 3 years later only 31% households were still using the filters (Brown et al, 2007)
• Pasteurisation in Kenya; 4 years later only 30% continued to pasteurise their water (Iijima et al, 2001)
• Programme promoting POU water disinfectant in Guatemala 1 year later; repeated use among only 5% of households from original trials (Luby et al, 2008).
• Water filters in Bolivia; compliance 67%; assessment made 4 months after trial ended (Clasen et al, 2006)
The Campbell Collaboration www.campbellcollaboration.org www.3ieimpact.orgAuthor name
Sustainability 3: lack of WTP
In Kenya, access to free chlorine increased uptake to over 60 percent, whereas coupons for even a 50 percent discount had a minimal effect
The Campbell Collaboration www.campbellcollaboration.org
– Point of use water treatment has large health effect (community-level doesn’t)
– But challenge is to ensure sustained proper use
– Any water supply intervention not taking into account this demand element should be questioned
So the systematic review tells us that:
The Campbell Collaboration www.campbellcollaboration.org
A quick word on the Campbell Collaboration• Coordinating groups (CGs) for
– Crime and Justice– Education– International development– Social welfare
• CGs manage editorial process– Three stage process: title, protocol, review– Any team can submit proposed title
• All published in Campbell Library, managed by Secretariat in Oslo
The Campbell Collaboration www.campbellcollaboration.org
In summary
• Rigorous evidence matters• High quality systematic reviews sort out what is
rigorous and what is not• And synthesize the evidence in policy-relevant ways• - telling us what works and why
• Make them and use them!!
The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org
Thank you
Visit our websitewww.campbellcollaboration.org
Follow us on Twitter @C2update & Facebook