The$Cultural$Evolu8on$of$Ineffec8ve$Medicine$ · 2015-12-04 · THE$IDEA$ THE$DATA$...
1
THE IDEA THE DATA THE IMPLICATIONS The Cultural Evolu8on of Ineffec8ve Medicine Mícheál de Barra 1 , Pontus Strimling & Kimmo Eriksson Everyone who is treated health outcome (+ive = health improved) Frequency −60 −40 −20 0 20 40 60 0 20 60 100 People who tell others about treatment health outcome (+ive = health improved) Frequency −60 −40 −20 0 20 40 60 0 20 60 100 Cumula8ve cultural evolu8on creates excellent solu8ons to many of life's problems. Hun8ng tools, s8tched clothing, protec8ve shelters and agricultural techniques show a gradual ratche8ng up in complexity, efficiency and u8lity (Richerson and Boyd 2005). But in another domains of life the process of cultural evolu8on rarely leads to adap8ve behavior. Most notably, tradi8onal, alterna8ve and historical medical treatments are typically ineffec8ve and oYen harmful (Woo[on 2006). So why do harmful or neutral treatments achieve cultural success? The outcome of a disease is determined by many factors, and so when a treatment is applied there will be a distribu8on of outcomes: some people will get be[er and some people will get worse. If the people with be[er outcomes are more inclined to share their experience with others, then a treatment may retain a posi8ve reputa8on despite a nega8ve effect. If this is the case, the reputa8on of medical treatments will be more posi8ve than the clinical evidence warrants. We tested this idea using diet books. 1,360 people had reviewed the Atkins diet book on amazon.com, and 540 of these state both an exact weight change and a 8me period over which this change happened (e.g. "OMG I lost 20lb in 3 months this diet is fantas8c"). These reviews allow an es8ma8on of the diets realworld reputa8on. Four clinical trials have examined weight change on the Atkins diet or similar and recorded weight change at mul8ple 8me points. Despite somewhat different interven8ons and samples, these studies show a consistent pa[ern of moderate loss. Fig 2 plots weight loss at each 8me period and shows that the average weight loss reported in amazon.com reviews of the Atkins diet are consistently larger than seen in clinical trials. after 2 months 0 5 10 15 20 25 (n=72) after 6 months (n=71) after 12 months (n=69) −40 −30 −20 −10 0 10 0 5 10 15 20 25 (n=81) −40 −30 −20 −10 0 10 (n=47) −40 −30 −20 −10 0 10 (n=42) weight change (kg) frequency (people) amazon.com clinical trial Atkins diet debiased 17 day−diet debiased condition number of people 0 10 20 30 40 50 Picked Atkins Picked 17−day diet Biased repor8ng will influence cultural evolu8on to the extent that the reputa8on influences subsequent decisions. We performed an online experiment where each par8cipant saw a picture of The Atkins Diet book plus three reviews, and The 17day Diet plus three reviews. In one condi8on the Atkins book reviews were “debiased” by (a) drawing reviews from a sample of reviews with an average of 3.5 stars (this is the average sa8sfac8on ra8ng given to diets in a longitudinal study by Baldwin et al. 2009) and (b) changing the weight change to the mean weight change at that 8me point as measured in clinical trials. In the other other condi8on, the 17day Diet was debiased using the same procedure. Order of diet presenta8on was counterbalanced. Fig 4. shows that par8cipants were significantly less likely to pick the pick the debiased diet. vs, Gardner et al. 2007 shared individual level data from their clinical trial of 311 women, including 77 on the Atkins diet. The interven8on entailed gelng the Atkins Diet Book plus 8 weekly mee8ngs with a die8cian to discuss the book and progress. Weight change was significantly larger in the amazon.com reviews than in the clinical trial (ps << .01). At 6 months, 27% of clinical trial par8cipants and 88% of amazon.com reviewers experienced weight loss of 10kg or more. Larger variance in clinical outcome was associated with a larger bias. The online reputed benefit of weightloss diets is larger than its real effect, most likely because people with typical or poorer outcomes are less likely to tell others about their experiences. Thus the realworld reputa8on of medical treatments may be subject to a publica8on bias akin to that seen in science (Easterbrook et al. 1991). This kind of bias is more likely to operate when: (a) treatment depends on wordofmouth reputa8on (b) treated individuals with poor outcomes can remain “invisible”, and (c) outcomes are a func8on of many factors, ie. there is a broad distribu8on of outcomes. When this is the case, ineffec8ve or even harmful treatments may maintain a posi8ve reputa8on. This may partly explain the persistence and prevalence of bad medicine. Man being harmed by a doctor: 1860s bloodlelng photo 2 . Fig 1. Imagined data illustra8ng how an ineffec8ve treatments can seem beneficial when people with posi8ve outcomes are more inclined to tell others about experience. Red lines indicate mean. Fig 2. Average weight loss on Atkins diet reported in four clinical trials and on amazon.com. Amazon data points calculated by averaging reviews diets of similar dura8on. 0 2 4 6 8 10 12 −30 −25 −20 −15 −10 −5 0 duration of diet (months) mean change in weight (kg) amazon.com Shai et al. 2008 Gardner et al. 2007 Truby et al. 2006 Foster et al. 2003 Fig 3. Comparison of weight change in Gardner et al. 2007 (top row) and amazon.com reviews (bo[om row) at three 8me points. Red lines indicate mean. Fig 4. Which diet would you choose? Results of experiment where par8cipants choose between a diet with typical reviews or a diet with “debiased” reviews. Atkins, R. C. (2002). Dr. Atkins' new diet revolu8on , M. Evans. Baldwin, A. S., A. J. Rothman, et al. (2009). "Sa8sfac8on with weight loss: examining the longitudinal covaria8on between people's weightlossrelated outcomes and experiences and their sa8sfac8on." Ann Behav Med 38(3): 21324. Easterbrook, P. J., J. A. Berlin, et al. (1991). "Publica8on bias in clinical research." Lancet 337(8746): 86772. Foster, G. D., H. R. Wya[, et al. (2003). "A randomized trial of a lowcarbohydrate diet for obesity." N Engl J Med 348(21): 208290. Gardner, C. D., A. Kiazand, et al. (2007). "Comparison of the Atkins, Zone, Ornish, and LEARN diets for change in weight and related risk factors among overweight premenopausal women: the A TO Z Weight Loss Study: a randomized trial." JAMA 297(9): 96977. Moreno, M. (2011). The 17 Day Diet: A Doctor's Plan Designed for Rapid Results , Free Press. Richerson, P. J. and R. Boyd (2005). Not by genes alone: How culture transformed human evolu8on . Chicago, IL, US, University of Chicago Press. Shai, I., D. Schwarzfuchs, et al. (2008). "Weight loss with a lowcarbohydrate, Mediterranean, or lowfat diet." N Engl J Med 359(3): 22941. Truby, H., S. Baic, et al. (2006). "Randomised controlled trial of four commercial weight loss programmes in the UK: ini8al findings from the BBC "diet trials"." BMJ 332(7553): 130914. Woo[on, D. (2006). Bad medicine: doctors doing harm since Hippocrates , Oxford University Press, USA. 1 Contact author at [email protected] 2 Image source: www.burnsarchive.com/ via wikipedia.
Transcript of The$Cultural$Evolu8on$of$Ineffec8ve$Medicine$ · 2015-12-04 · THE$IDEA$ THE$DATA$...
THE IDEA
THE DATA
THE IMPLICATIONS
The Cultural Evolu8on of Ineffec8ve Medicine Mícheál de Barra1, Pontus Strimling & Kimmo Eriksson
Everyone who is treated
health outcome (+ive = health improved)
Freq
uenc
y
−60 −40 −20 0 20 40 60
020
6010
0
People who tell others about treatment
health outcome (+ive = health improved)
Freq
uenc
y
−60 −40 −20 0 20 40 60
020
6010
0
Cumula8ve cultural evolu8on creates excellent solu8ons to many of life's problems. Hun8ng tools,
s8tched clothing, protec8ve shelters and agricultural techniques show a gradual ratche8ng up in
complexity, efficiency and u8lity (Richerson and Boyd 2005). But in another domains of life the
process of cultural evolu8on rarely leads to adap8ve behavior. Most notably, tradi8onal, alterna8ve
and historical medical treatments are typically ineffec8ve and oYen harmful (Woo[on 2006).
So why do harmful or neutral treatments achieve cultural success? The outcome of a disease is
determined by many factors, and so when a treatment is applied there will be a distribu8on of
outcomes: some people will get be[er and some people will get worse. If the people with be[er
outcomes are more inclined to share their experience with others, then a treatment may retain a
posi8ve reputa8on despite a nega8ve effect. If this is the case, the reputa8on of medical treatments
will be more posi8ve than the clinical evidence warrants.
We tested this idea using diet books. 1,360 people had reviewed the Atkins diet book on
amazon.com, and 540 of these state both an exact weight change and a 8me period over which this
change happened (e.g. "OMG I lost 20lb in 3 months -‐ this diet is fantas8c"). These reviews allow an
es8ma8on of the diets real-‐world reputa8on. Four clinical trials have examined weight change on the
Atkins diet or similar and recorded weight change at mul8ple 8me points. Despite somewhat
different interven8ons and samples, these studies show a consistent pa[ern of moderate loss. Fig 2
plots weight loss at each 8me period and shows that the average weight loss reported in amazon.com
reviews of the Atkins diet are consistently larger than seen in clinical trials.
after 2 months
05
1015
2025
(n=72)
after 6 months
(n=71)
after 12 months
(n=69)
−40 −30 −20 −10 0 10
05
1015
2025
(n=81)
−40 −30 −20 −10 0 10
(n=47)
−40 −30 −20 −10 0 10
(n=42)
weight change (kg)
frequ
ency
(peo
ple)
a
maz
on.c
omcl
inic
al tr
ial
Atkins diet debiased 17 day−diet debiased
condition
num
ber o
f peo
ple
010
2030
4050
Picked AtkinsPicked 17−day diet
Biased repor8ng will influence cultural evolu8on to the extent that the reputa8on influences subsequent decisions. We performed
an online experiment where each par8cipant saw a picture of The Atkins Diet book plus three reviews, and The 17-‐day Diet plus
three reviews. In one condi8on the Atkins book reviews were “debiased” by (a) drawing reviews from a sample of reviews with an
average of 3.5 stars (this is the average sa8sfac8on ra8ng given to diets in a longitudinal study by Baldwin et al. 2009) and (b)
changing the weight change to the mean weight change at that 8me point as measured in clinical trials. In the other other
condi8on, the 17-‐day Diet was debiased using the same procedure. Order of diet presenta8on was counterbalanced. Fig 4. shows
that par8cipants were significantly less likely to pick the pick the debiased diet.
vs,
Gardner et al. 2007 shared individual level data from their clinical trial of 311
women, including 77 on the Atkins diet. The interven8on entailed gelng the
Atkins Diet Book plus 8 weekly mee8ngs with a die8cian to discuss the book
and progress. Weight change was significantly larger in the amazon.com
reviews than in the clinical trial (ps << .01). At 6 months, 27% of clinical trial
par8cipants and 88% of amazon.com reviewers experienced weight loss of
10kg or more. Larger variance in clinical outcome was associated with a larger
bias.
The online reputed benefit of weight-‐loss diets is larger than its real effect, most likely because
people with typical or poorer outcomes are less likely to tell others about their experiences. Thus the
real-‐world reputa8on of medical treatments may be subject to a publica8on bias akin to that seen in
science (Easterbrook et al. 1991). This kind of bias is more likely to operate when: (a) treatment
depends on word-‐of-‐mouth reputa8on (b) treated individuals with poor outcomes can remain
“invisible”, and (c) outcomes are a func8on of many factors, ie. there is a broad distribu8on of
outcomes. When this is the case, ineffec8ve or even harmful treatments may maintain a posi8ve
reputa8on. This may partly explain the persistence and prevalence of bad medicine.
Man being harmed by a doctor: 1860s bloodlelng photo2.
Fig 1. Imagined data illustra8ng how an ineffec8ve treatments can seem beneficial when people with posi8ve outcomes are more inclined to tell others about experience. Red lines indicate mean.
Fig 2. Average weight loss on Atkins diet reported in four clinical trials and on amazon.com. Amazon data points calculated by averaging reviews diets of similar dura8on.
0 2 4 6 8 10 12
−30
−25
−20
−15
−10
−50
duration of diet (months)
mea
n ch
ange
in w
eigh
t (kg
)
amazon.comShai et al. 2008Gardner et al. 2007Truby et al. 2006Foster et al. 2003
Fig 3. Comparison of weight change in Gardner et al. 2007 (top row) and amazon.com reviews (bo[om row) at three 8me points. Red lines indicate mean.
Fig 4. Which diet would you choose? Results of experiment where par8cipants choose between a diet with typical reviews or a diet with “debiased” reviews.
Atkins, R. C. (2002). Dr. Atkins' new diet revolu8on, M. Evans.
Baldwin, A. S., A. J. Rothman, et al. (2009). "Sa8sfac8on with weight loss: examining the longitudinal covaria8on between
people's weight-‐loss-‐related outcomes and experiences and their sa8sfac8on." Ann Behav Med 38(3): 213-‐24.
Easterbrook, P. J., J. A. Berlin, et al. (1991). "Publica8on bias in clinical research." Lancet 337(8746): 867-‐72.
Foster, G. D., H. R. Wya[, et al. (2003). "A randomized trial of a low-‐carbohydrate diet for obesity." N Engl J Med 348(21):
2082-‐90.
Gardner, C. D., A. Kiazand, et al. (2007). "Comparison of the Atkins, Zone, Ornish, and LEARN diets for change in weight and
related risk factors among overweight premenopausal women: the A TO Z Weight Loss Study: a randomized trial." JAMA
297(9): 969-‐77.
Moreno, M. (2011). The 17 Day Diet: A Doctor's Plan Designed for Rapid Results, Free Press.
Richerson, P. J. and R. Boyd (2005). Not by genes alone: How culture transformed human evolu8on. Chicago, IL, US, University
of Chicago Press.
Shai, I., D. Schwarzfuchs, et al. (2008). "Weight loss with a low-‐carbohydrate, Mediterranean, or low-‐fat diet." N Engl J Med
359(3): 229-‐41.
Truby, H., S. Baic, et al. (2006). "Randomised controlled trial of four commercial weight loss programmes in the UK: ini8al
findings from the BBC "diet trials"." BMJ 332(7553): 1309-‐14.
Woo[on, D. (2006). Bad medicine: doctors doing harm since Hippocrates, Oxford University Press, USA. 1 Contact author at [email protected] 2 Image source: www.burnsarchive.com/ via wikipedia.
