A Skeptic’s Guide to Reading the Medical Literature
description
Transcript of A Skeptic’s Guide to Reading the Medical Literature
A Skeptic’s Guide A Skeptic’s Guide to Reading the to Reading the
Medical LiteratureMedical Literature
Joe Lex, MD, FACEP, MAAEMJoe Lex, MD, FACEP, MAAEMClinical Professor of Emergency MedicineClinical Professor of Emergency Medicine
Temple University School of MedicineTemple University School of MedicinePhiladelphia, PA USAPhiladelphia, PA USA
[email protected] @JoeLex5 @JoeLex5
Peer ReviewPeer Review
Peer Review = Gold StandardPeer Review = Gold Standard
SlowSlow ExpensiveExpensive IneffectiveIneffective ““Somewhat of a lottery”Somewhat of a lottery” Prone to bias and abuseProne to bias and abuse Hopeless at spotting error and Hopeless at spotting error and
fraudfraud
Peer Review = Gold StandardPeer Review = Gold Standard
“…“…peer review is impossible to peer review is impossible to define in operational terms…Peer define in operational terms…Peer review is thus like poetry, love, or review is thus like poetry, love, or justice.”justice.”
Smith R. J R Soc Med. 2006 Smith R. J R Soc Med. 2006 Apr;99(4):178-82. Apr;99(4):178-82.
Peer Review = Gold StandardPeer Review = Gold Standard
““When I was editor of the BMJ I was When I was editor of the BMJ I was challenged by two of the cleverest challenged by two of the cleverest researchers in Britain to publish an researchers in Britain to publish an issue of the journal comprised only issue of the journal comprised only of papers that had failed peer review of papers that had failed peer review and see if anybody noticed. I wrote and see if anybody noticed. I wrote back ‘How do you know I haven’t back ‘How do you know I haven’t already done it?’”already done it?’”
Richard Smith – Editor, BMJRichard Smith – Editor, BMJ
Peer Review = Gold StandardPeer Review = Gold Standard
Two reviewers, same paperTwo reviewers, same paper
Reviewer A:Reviewer A: I found this paper an I found this paper an extremely muddled paper with a extremely muddled paper with a large number of deficits large number of deficits
Reviewer B:Reviewer B: It is written in a clear It is written in a clear style and would be understood by style and would be understood by any readerany reader
Richard Smith – Editor, BMJRichard Smith – Editor, BMJ
Peer Review = Gold StandardPeer Review = Gold Standard
Benefits: ???Benefits: ??? Editors are convinced that it is Editors are convinced that it is
invaluableinvaluable ““If (peer review) was a drug it If (peer review) was a drug it
would never get onto the market.”would never get onto the market.”
Drummond Rennie, Drummond Rennie, Deputy Editor of JAMADeputy Editor of JAMA
Positive-Outcome BiasPositive-Outcome Bias
Dummy article, 210 reviewersDummy article, 210 reviewers Same except for conclusionSame except for conclusion
– Half got “positive conclusion”Half got “positive conclusion”
– Half got “no difference”Half got “no difference” Intentional errors includedIntentional errors included Methods section sameMethods section same
Emerson GB et al. Arch Int Med 2010, 170: 1934 Emerson GB et al. Arch Int Med 2010, 170: 1934
Positive-Outcome BiasPositive-Outcome Bias
Emerson GB et al. Arch Int Med 2010, 170: 1934 Emerson GB et al. Arch Int Med 2010, 170: 1934
Positive Positive outcomeoutcome
Neutral Neutral outcomeoutcome
Recommend Recommend publicationpublication 97.3%97.3% 80.0%80.0%
Errors Errors identifiedidentified 0.410.41 0.850.85
Methods Methods validity scorevalidity score 8.248.24 7.537.53
Positive-Outcome BiasPositive-Outcome Bias
““Compromise(s) integrity of the Compromise(s) integrity of the literature”literature”– Inflates treatment effects when Inflates treatment effects when
included in meta-analysisincluded in meta-analysis
Emerson GB et al. Arch Int Med 2010, 170: 1934 Emerson GB et al. Arch Int Med 2010, 170: 1934
Peer Review Misses ThingsPeer Review Misses Things
Fake manuscriptFake manuscript Contained 10 major and 13 minor Contained 10 major and 13 minor
errorserrors Sent to 262 reviewersSent to 262 reviewers 203 returned203 returned
Baxt WB et al. Ann Emerg Med 1998:32, 310 Baxt WB et al. Ann Emerg Med 1998:32, 310
Peer Review Misses ThingsPeer Review Misses Things
15 ‘accept’15 ‘accept’– 17% major, 12% minor17% major, 12% minor
117 ‘reject’117 ‘reject’– 39% major, 21% minor39% major, 21% minor
67 ‘revise’67 ‘revise’– 30% major, 22% minor30% major, 22% minor
Baxt WB et al. Ann Emerg Med 1998:32, 310 Baxt WB et al. Ann Emerg Med 1998:32, 310
Peer Review Misses ThingsPeer Review Misses Things
68% did not recognize that 68% did not recognize that conclusion was not supported by conclusion was not supported by datadata
Failed to identify 68% of major Failed to identify 68% of major errorserrors
Baxt WB et al. Ann Emerg Med 1998:32, 310 Baxt WB et al. Ann Emerg Med 1998:32, 310
Cochrane AnalysisCochrane Analysis
““We could not identify any We could not identify any methodologically convincing methodologically convincing studies assessing the core effects studies assessing the core effects of peer review”of peer review”
““Little empirical evidence to support Little empirical evidence to support use of editorial peer review to use of editorial peer review to ensure quality.”ensure quality.”
Jefferson T, et al. Cochrane DatabaseJefferson T, et al. Cochrane Database
Watch out for…Watch out for…
……strong strong associationsassociations
……independent independent predictorspredictors
……citation biascitation bias……amplificationamplification……inventioninvention……composite composite
endpointsendpoints
……workup biasworkup bias……spectrum biasspectrum bias……referral biasreferral bias……surrogate markerssurrogate markers……publication biaspublication bias……absolute vs absolute vs
relative riskrelative risk
Strong AssociationStrong Association
There is a “strong association” There is a “strong association” between hanging out in bars and between hanging out in bars and developing lung cancerdeveloping lung cancer
People who hang out in bars tend People who hang out in bars tend to smoke and drink a lotto smoke and drink a lot
Does hanging out in bars cause Does hanging out in bars cause lung cancer?lung cancer?
Strong AssociationStrong Association
There is a “strong association” There is a “strong association” between the rooster crowing at between the rooster crowing at 06:15 and the sun rising at 06:2006:15 and the sun rising at 06:20
Does the rooster’s crow cause the Does the rooster’s crow cause the sun to rise?sun to rise?
Strong AssociationStrong Association
No correlation with…No correlation with…
……sensitivitysensitivity
……specificityspecificity
……accuracyaccuracy
……usefulnessusefulness
Independent PredictorIndependent Predictor
If patients are matched in all other If patients are matched in all other parameters, the group with parameters, the group with abnormal test results will have abnormal test results will have more outcome events than the more outcome events than the group with normal test resultsgroup with normal test results
Independent PredictorIndependent Predictor
In the ICU, three organ failure at In the ICU, three organ failure at three days carries a 97% mortality three days carries a 97% mortality raterate
So who don’t I have to treat, since So who don’t I have to treat, since it’s so futile?it’s so futile?
Predictors & AssociationsPredictors & Associations
Does not mean:Does not mean:““Your test is negative, you can go Your test is negative, you can go
home now.”home now.”““Your test is positive, you need to Your test is positive, you need to
stay and have more studies.”stay and have more studies.” NoNo valuevalue in making diagnostic or in making diagnostic or
therapeutic decisionstherapeutic decisions
Negative Predictive ValueNegative Predictive Value
Probability of no disease among Probability of no disease among patients with a negative testpatients with a negative test
NPV = TN / (TN + FN)NPV = TN / (TN + FN) Denominator is Denominator is number of number of
patients who test negativepatients who test negative So if disease prevalence low…So if disease prevalence low…
Negative Predictive ValueNegative Predictive Value
NPV does NPV does notnot tell you test value tell you test value If disease prevalence 5%, coin flip If disease prevalence 5%, coin flip
has NPV = 95%has NPV = 95% If disease prevalence 85%, coin If disease prevalence 85%, coin
flip has PPV = 85%flip has PPV = 85%
Citation BiasCitation Bias
Citation biasCitation bias: not citing papers : not citing papers that refute or weaken beliefthat refute or weaken belief
AmplificationAmplification: expand belief : expand belief system by citing papers that system by citing papers that present no data addressing itpresent no data addressing it
InventionInvention: hypothesis converted : hypothesis converted into fact through citation aloneinto fact through citation alone
Redundant PublicationRedundant Publication
Ondansetron in post-op vomitingOndansetron in post-op vomiting 84 trials with 11,980 patients84 trials with 11,980 patients On closer inspection: 70 trials with On closer inspection: 70 trials with
8645 patients8645 patients 17% duplicate publications17% duplicate publications 28% inflation of patients28% inflation of patients
Tramer MR et al. BMJ 1997:315;635-640.Tramer MR et al. BMJ 1997:315;635-640.
Redundant PublicationRedundant Publication
Four identical trials published with Four identical trials published with different authorsdifferent authors
Duplicated publications most likely Duplicated publications most likely to have positive resultsto have positive results
NNT in nonduplicated trials: 9.5NNT in nonduplicated trials: 9.5 NNT in duplicated trials: 3.9NNT in duplicated trials: 3.9
Tramer MR et al. BMJ 1997:315;635-640.Tramer MR et al. BMJ 1997:315;635-640.
Multiple HypothesesMultiple Hypotheses
Multiple HypothesesMultiple Hypotheses
10,674,945 Ontario residents 10,674,945 Ontario residents aged between 18 and 100 years aged between 18 and 100 years – Derivation cohort 5,337,472 Derivation cohort 5,337,472
– Validation cohort 5,337,473Validation cohort 5,337,473
895 diagnoses for which patients 895 diagnoses for which patients had emergent / urgent had emergent / urgent hospitalizationhospitalization
Born Under a Bad SignBorn Under a Bad Sign
Goal: identify two diagnoses for Goal: identify two diagnoses for each astrological sign for which each astrological sign for which the probability of hospitalization the probability of hospitalization was statistically significantly was statistically significantly greater compared to residents greater compared to residents born under the remaining 11 signsborn under the remaining 11 signs
Born Under a Bad SignBorn Under a Bad Sign
Found 72 such associationsFound 72 such associations Number of diagnoses ranged from Number of diagnoses ranged from
2 (Scorpio) to 10 (Taurus), with a 2 (Scorpio) to 10 (Taurus), with a mean of 6 diagnoses for each mean of 6 diagnoses for each astrological signastrological sign
The P-values for 72 significant The P-values for 72 significant associations: 0.0003 to 0.0488associations: 0.0003 to 0.0488
Born Under a Bad SignBorn Under a Bad Sign
Validation cohort: residents born Validation cohort: residents born under sign of Leo had significantly under sign of Leo had significantly higher probability of higher probability of hospitalization due to GI hospitalization due to GI hemorrhage compared to other hemorrhage compared to other residents of Ontarioresidents of Ontario– Relative risk = 1.15 (p = 0.0483)Relative risk = 1.15 (p = 0.0483)
Born Under a Bad SignBorn Under a Bad Sign
Validation cohort: residents born Validation cohort: residents born under sign of Sagittarius had under sign of Sagittarius had significantly higher probability of significantly higher probability of hospitalization for fractures of hospitalization for fractures of humerus compared to residents humerus compared to residents born under the remaining 11 born under the remaining 11 astrological signsastrological signs– Relative risk = 1.38 (p = 0.0125)Relative risk = 1.38 (p = 0.0125)
If an ineffective therapy is tested If an ineffective therapy is tested for 20 indications, by chance it is for 20 indications, by chance it is likely to demonstrate a significant likely to demonstrate a significant effect for at least one of theseeffect for at least one of these
Do the Math!!Do the Math!!
Ioannidis JP, et al. BMJ. 2010 Sep 13;341:c4875.Ioannidis JP, et al. BMJ. 2010 Sep 13;341:c4875.
If 10 outcomes are tested for each If 10 outcomes are tested for each indication, a statistically significant indication, a statistically significant effect will be observed for at least effect will be observed for at least one outcome for nearly half of the one outcome for nearly half of the indicationsindications
Do the Math!!Do the Math!!
Ioannidis JP, et al. BMJ. 2010 Sep 13;341:c4875.Ioannidis JP, et al. BMJ. 2010 Sep 13;341:c4875.
In 91 trials discontinued early, the In 91 trials discontinued early, the true effect was, on average, only true effect was, on average, only 70% of that suggested by trials 70% of that suggested by trials
It was <50% of perceived effect It was <50% of perceived effect when trials were discontinued when trials were discontinued after occurrence of <200 eventsafter occurrence of <200 events
Do the Math!!Do the Math!!
Ioannidis JP, et al. BMJ. 2010 Sep 13;341:c4875.Ioannidis JP, et al. BMJ. 2010 Sep 13;341:c4875.
Workup BiasWorkup Bias
In routine care the diagnostic In routine care the diagnostic workup of a patient is by definition workup of a patient is by definition determined by previous test determined by previous test resultsresults
Positive screening test is referred Positive screening test is referred to receive verification of diagnosis to receive verification of diagnosis by “gold standard”by “gold standard”
Workup BiasWorkup Bias
Reference test is always Reference test is always interpreted with knowledge of interpreted with knowledge of preceding test information preceding test information
Example: treadmill stress testExample: treadmill stress test– Family practitioner’s office: Family practitioner’s office:
sensitivity 40%, specificity 85%sensitivity 40%, specificity 85%
– Cardiologist’s office: sensitivity Cardiologist’s office: sensitivity 70%, specificity 70%70%, specificity 70%
Spectrum BiasSpectrum Bias
Diagnostic test has different Diagnostic test has different sensitivities or specificities in sensitivities or specificities in patients with different clinical patients with different clinical manifestations of the disease for manifestations of the disease for which the test is intended which the test is intended
Spectrum BiasSpectrum Bias
Patient obviously has disease: Patient obviously has disease: test is positivetest is positive
Patient obviously does not have Patient obviously does not have disease: test is negativedisease: test is negative
““I don’t know if the patient has the I don’t know if the patient has the disease” – test in betweendisease” – test in between– Example: BNPExample: BNP
Spectrum BiasSpectrum Bias
If pretest probability <5%, you’re If pretest probability <5%, you’re more accurate than BNPmore accurate than BNP
If pretest probability >95%, you’re If pretest probability >95%, you’re more accurate than BNPmore accurate than BNP
The “In Between” GroupThe “In Between” Group
Sensitivity: 88% Sensitivity: 88%
Specificity: 55%Specificity: 55%
Spectrum BiasSpectrum Bias
If the serum BNP is…If the serum BNP is…
……>500 pg/mL >500 pg/mL rule in CHF rule in CHF
……<100 pg/mL <100 pg/mL rule out CHF if low rule out CHF if low pretest probabilitypretest probability
……100 – 500 pg/mL: not sensitive, 100 – 500 pg/mL: not sensitive, not specific, and not very helpfulnot specific, and not very helpful
Referral BiasReferral Bias
Sanford GuideSanford Guide says: “80% of cat says: “80% of cat bites get infected”bites get infected”
Dog bite: self-referral because of Dog bite: self-referral because of wound size, rabies worry, etc.wound size, rabies worry, etc.
Cat bite: “It’s infected”Cat bite: “It’s infected” Headache in oncologist’s officeHeadache in oncologist’s office Step on a nailStep on a nail
Other BiasesOther Biases
Length biasLength bias: indolent disease is : indolent disease is more likely to be detected in a more likely to be detected in a screening program than screening program than aggressive disease, leading to aggressive disease, leading to apparent improved outcome apparent improved outcome
Other BiasesOther Biases
Lead-time biasLead-time bias: survival of a : survival of a screened population is measured screened population is measured from the date of screening, from the date of screening, whereas survival of an whereas survival of an unscreened population is unscreened population is measured from detection of measured from detection of symptomatic diseasesymptomatic disease
Absolute vs relative Absolute vs relative
Can statins prevent VTE?Can statins prevent VTE? JUPITER trial says “yes!!”JUPITER trial says “yes!!” ““Patients taking rosuvastatin Patients taking rosuvastatin
((CrestorCrestor) 20 mg/day have ) 20 mg/day have 43% 43% lower risklower risk of DVT or PE” of DVT or PE”
Absolute vs relative Absolute vs relative
These patients relatively healthy These patients relatively healthy ABSOLUTEABSOLUTE risk drops from about risk drops from about
0.7% to 0.4%0.7% to 0.4% Treat 333 patients for one year to Treat 333 patients for one year to
prevent one DVT or PE prevent one DVT or PE
Surrogate EndpointsSurrogate Endpoints
Some surrogate end points may Some surrogate end points may be part of disease or risk factor be part of disease or risk factor
Sometimes surrogate endpoint Sometimes surrogate endpoint confused with diseaseconfused with disease
Example: sitagliptin enhances Example: sitagliptin enhances incretins and enhances incretins and enhances physiological glucose controlphysiological glucose control
DrugDrug SurrogateSurrogate EndpointEndpoint
EncainideEncainide ↓ ↓ premature premature depolarizationdepolarization ↑ ↑ mortalitymortality
DoxazosinDoxazosin ↓ ↓ BPBP ↑ ↑ mortalitymortality
EstrogenEstrogen ↓ ↓ cholesterolcholesterol ↑ ↑ cancer, stroke, cancer, stroke, dementiadementia
TorcetrapidTorcetrapid ↓ ↓ cholesterolcholesterol ↑ ↑ mortalitymortality
NesiritideNesiritide ↓ ↓ wedge wedge pressurepressure ↑ ↑ mortalitymortality
FluorideFluoride ↑ ↑ bone mineral bone mineral densitydensity
↑ ↑ nonvertebral nonvertebral fracturesfractures
DrugDrug SurrogateSurrogate EndpointEndpoint
RosiglitazoneRosiglitazone ↓ ↓ S-HbAS-HbA1c1c↑ ↑ myocardial myocardial infarctioninfarction
Intensive Intensive treatment treatment Type II DMType II DM
↓ ↓ S-HbAS-HbA1c1c
<6%<6%↑ ↑ mortalitymortality
MilrinoneMilrinone ↑ ↑ cardiac cardiac contractilitycontractility ↑ ↑ mortalitymortality
IbopamineIbopamine ↑ ↑ cardiac cardiac contractilitycontractility ↑ ↑ mortalitymortality
EpoprostenolEpoprostenol ↑ ↑ cardiac cardiac contractilitycontractility ↑ ↑ mortalitymortality
DrugDrug SurrogateSurrogate EndpointEndpoint
TobutamideTobutamide ↓ ↓ blood sugarblood sugar ↑ ↑ mortalitymortality
PhenforrminPhenforrmin ↓ ↓ blood sugarblood sugar ↑ ↑ mortalitymortality
ErythropoeitinErythropoeitin ↑ ↑ hemoglobin in hemoglobin in renal failurerenal failure ↑ ↑ mortalitymortality
AprotininAprotinin ↓ ↓ blood loss in blood loss in cardiac surgerycardiac surgery ↑ ↑ mortalitymortality
MetoprololMetoprolol ↓ ↓ perioperative perioperative ischaemiaischaemia ↑ ↑ mortalitymortality
Need POEM, Not DOENeed POEM, Not DOE
ED Patients
Experimental: New Test
Control: Standard Tests
Randomize Randomize
Outcomes: Morbidity, Mortality, Admissions, Length of Stay, Cost
CURE Trial: ClopidogrelCURE Trial: Clopidogrel
>12,500 patients for 3 – 12 >12,500 patients for 3 – 12 monthsmonths
Inclusion criteria: USA + Inclusion criteria: USA + cardiac cardiac biomarker or dynamic ECG biomarker or dynamic ECG changechange
CURE Trial: ClopidogrelCURE Trial: Clopidogrel
Primary end point: composite of Primary end point: composite of cardiovascular death cardiovascular death oror MI MI oror nonfatal strokenonfatal stroke
Placebo + aspirin: 11.4%Placebo + aspirin: 11.4%
Clopidogrel + aspirin: 9.3% Clopidogrel + aspirin: 9.3%
CURE Trial: ClopidogrelCURE Trial: Clopidogrel
Ads says “19.2% reduction in Ads says “19.2% reduction in death or cardiovascular event”death or cardiovascular event”
This is RELATIVE reductionThis is RELATIVE reduction Absolute reduction: 2.1%Absolute reduction: 2.1% NNT to prevent one death NNT to prevent one death oror MI MI
oror stroke: 48 stroke: 48
ClopidogrelClopidogrel
n = 6259n = 6259
PlaceboPlacebo
n = 6303n = 6303
P P valuevalue
Any 1Any 1oo or 2 or 2oo outcomeoutcome 16.5%16.5% 18.8%18.8% <0.001<0.001
CV deathCV death 5.1%5.1% 5.5%5.5% NSNS
MIMI 5.2%5.2% 6.7%6.7% <0.001<0.001
StrokeStroke
IschemiaIschemia
1.2%1.2%
8.7%8.7%
1.4%1.4%
9.3%9.3%
NSNS
NSNS
CURE Trial: ClopidogrelCURE Trial: Clopidogrel
CURE Trial: ClopidogrelCURE Trial: Clopidogrel
NNT to prevent one:NNT to prevent one:
……Cardiovascular death Cardiovascular death 250250
……Myocardial infarction Myocardial infarction 67 67
……StrokeStroke500500
……Refractory ischemiaRefractory ischemia 167167
CURE Trial: ClopidogrelCURE Trial: Clopidogrel
Adverse effects: major bleedAdverse effects: major bleed Aspirin only: 2.7%Aspirin only: 2.7% Aspirin + clopidogrel: 3.6%Aspirin + clopidogrel: 3.6% Relative increase: 33%Relative increase: 33% Absolute increase: 0.9%Absolute increase: 0.9% NNH: 110NNH: 110
CURE Trial: ClopidogrelCURE Trial: Clopidogrel
NNT to cause*NNT to cause*
Major bleedMajor bleed 100100
Life-threatening bleedLife-threatening bleed 250250
DiarrheaDiarrhea 91 91
PurpuraPurpura 62 62
ItchingItching 59 59
*compared to *compared to aspirinaspirin
Fibrinolysis and StrokeFibrinolysis and Stroke
Manufacturer-sponsored study of Manufacturer-sponsored study of IV alteplase in 821 patientsIV alteplase in 821 patients
Given 3 – 4.5 hours after onset of Given 3 – 4.5 hours after onset of ischemic stroke symptomsischemic stroke symptoms
Exclude from treatment: minor or Exclude from treatment: minor or rapidly improving, severe stroke, rapidly improving, severe stroke, prior stroke, diabetes, age >80prior stroke, diabetes, age >80
Hacke W, et al, NEJM 2008 Sep 25, 359(13):1317Hacke W, et al, NEJM 2008 Sep 25, 359(13):1317
Fibrinolysis and StrokeFibrinolysis and Stroke
AlteplaseAlteplase Controls Controls
Modified Rankin 0 – 1Modified Rankin 0 – 1 52.4%52.4% 45.2%45.2%
Intracerebral Intracerebral hemorrhage (ICH)hemorrhage (ICH) 27.0%27.0% 17.6%17.6%
Symptomatic ICHSymptomatic ICH 2.4%2.4% 0.2%0.2%
90-day mortality90-day mortality 7.7%7.7% 8.4%8.4%
Hacke W. et al, NEJM 2008 Sep 25, 359(13):1317Hacke W. et al, NEJM 2008 Sep 25, 359(13):1317
Fibrinolysis and StrokeFibrinolysis and Stroke
Fibrinolysis and StrokeFibrinolysis and Stroke
ECASS-III dichotomized the ECASS-III dichotomized the modified Rankin Scalemodified Rankin Scale
Instead of using 0-6Instead of using 0-6
0 – 1 was favorable0 – 1 was favorable
2 – 6 was unfavorable2 – 6 was unfavorable But 2 is slight disability…But 2 is slight disability…
……and 6 is deathand 6 is death
Fibrinolysis and StrokeFibrinolysis and Stroke
00 No symptoms at allNo symptoms at all11 Able to carry out all usual activitiesAble to carry out all usual activities22 Slight disability, but able to look after Slight disability, but able to look after
own affairs without assistanceown affairs without assistance33 Moderate disability; requires some help, Moderate disability; requires some help,
but walks without assistancebut walks without assistance44 Moderately severe disability; unable to Moderately severe disability; unable to
walk without assistance walk without assistance 55 Severe disability; bedridden, incontinentSevere disability; bedridden, incontinent66 DeadDead
Fibrinolysis and StrokeFibrinolysis and Stroke
00 No symptoms at allNo symptoms at all11 Able to carry out all usual activitiesAble to carry out all usual activities22 Slight disability, but able to look after Slight disability, but able to look after
own affairs without assistanceown affairs without assistance33 Moderate disability; requires some help, Moderate disability; requires some help,
but walks without assistancebut walks without assistance44 Moderately severe disability; unable to Moderately severe disability; unable to
walk without assistance walk without assistance 55 Severe disability; bedridden, incontinentSevere disability; bedridden, incontinent66 DeadDead
Fibrinolysis and StrokeFibrinolysis and Stroke
Now adjust the scaleNow adjust the scale
Make 0 – 2 favorableMake 0 – 2 favorable
Make 3 – 6 unfavorableMake 3 – 6 unfavorable
……and the advantage of IV rt-PA and the advantage of IV rt-PA over placebo goes away in the over placebo goes away in the intention-to-treat groupintention-to-treat group
The Australasian Journal of The Australasian Journal of Bone and Joint Medicine Bone and Joint Medicine
““Merck paid an undisclosed sum to Merck paid an undisclosed sum to Elsevier to produce several volumes of a Elsevier to produce several volumes of a publication that had the look of a peer-publication that had the look of a peer-reviewed medical journal, but contained reviewed medical journal, but contained only reprinted or summarized articles –only reprinted or summarized articles –most of which presented data favorable most of which presented data favorable to Merck products – that appeared to act to Merck products – that appeared to act solely as marketing tools with no solely as marketing tools with no disclosure of company sponsorship.”disclosure of company sponsorship.”
ElsevierElsevier
Elsevier also publishes...Elsevier also publishes...
……pseudoscientific pseudoscientific HomeopathyHomeopathy
……several “integrative medicine” several “integrative medicine” journals journals
……highly dubious highly dubious ExploreExplore, with , with articles on “distant healing” and articles on “distant healing” and food imbued with “intent”food imbued with “intent”
What Gets Published??What Gets Published??
Why Articles Are FalseWhy Articles Are False
1.1. The smaller the studies, the less The smaller the studies, the less likely the research findings are to likely the research findings are to be truebe true
2.2. The smaller the effect, the less The smaller the effect, the less likely the research findings are to likely the research findings are to be truebe true
Ioannidis JPA. PLoS Med. 2005;2(8):e124Ioannidis JPA. PLoS Med. 2005;2(8):e124
Why Articles Are FalseWhy Articles Are False
3.3. The greater the number and the The greater the number and the lesser the selection of tested lesser the selection of tested relationships, the less likely the relationships, the less likely the research findings are to be trueresearch findings are to be true
Ioannidis JPA. PLoS Med. 2005;2(8):e124Ioannidis JPA. PLoS Med. 2005;2(8):e124
Why Articles Are FalseWhy Articles Are False
4.4. The greater the flexibility in The greater the flexibility in designs, definitions, outcomes, designs, definitions, outcomes, and analytical modes, the less and analytical modes, the less likely the research findings are to likely the research findings are to be truebe true
Ioannidis JPA. PLoS Med. 2005;2(8):e124Ioannidis JPA. PLoS Med. 2005;2(8):e124
Why Articles Are FalseWhy Articles Are False
5.5. The greater the financial and The greater the financial and other interests, the less likely the other interests, the less likely the research findings are to be trueresearch findings are to be true
Ioannidis JPA. PLoS Med. 2005;2(8):e124Ioannidis JPA. PLoS Med. 2005;2(8):e124
Why Articles Are FalseWhy Articles Are False
6.6. The hotter a scientific field (with The hotter a scientific field (with more scientific teams involved), more scientific teams involved), the less likely the research the less likely the research findings are to be truefindings are to be true
Ioannidis JPA. PLoS Med. 2005;2(8):e124Ioannidis JPA. PLoS Med. 2005;2(8):e124
Publication BiasPublication Bias
Negative study unlikely to be first Negative study unlikely to be first publishedpublished
Positive study published: many Positive study published: many people are early adopterspeople are early adopters
Once accepted as “standard,” Once accepted as “standard,” company has no incentive to company has no incentive to sponsor further studiessponsor further studies
Publication BiasPublication Bias
Turner EH, Matthews AM, Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Linardatos E, Tell RA, Rosenthal R.
Selective publication of Selective publication of antidepressant trials and its antidepressant trials and its influence on apparent efficacy.influence on apparent efficacy.
N Engl J Med. 2008 Jan N Engl J Med. 2008 Jan 17;358(3):252-60. 17;358(3):252-60.
Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60. Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60.
Publication BiasPublication Bias
74 FDA-registered studies on 74 FDA-registered studies on antidepressantsantidepressants
Represented 3449 participantsRepresented 3449 participants 37 studies viewed as positive and 37 studies viewed as positive and
were publishedwere published 1 study viewed as positive and 1 study viewed as positive and
was not publishedwas not published
Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60. Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60.
Publication BiasPublication Bias
74 FDA-registered studies on 74 FDA-registered studies on antidepressantsantidepressants
22 studies viewed as negative or 22 studies viewed as negative or questionable: not publishedquestionable: not published
11 studies viewed as negative or 11 studies viewed as negative or questionable: published in way questionable: published in way that conveyed positive outcomethat conveyed positive outcome
Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60. Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60.
Publication BiasPublication Bias
According to published literature: According to published literature: 94% of trials on antidepressant 94% of trials on antidepressant drugs are positivedrugs are positive
Actual data: 51% of trials on Actual data: 51% of trials on antidepressant drugs are positiveantidepressant drugs are positive
Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60. Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60.
Who Pays the Piper?Who Pays the Piper?
Who Pays the Piper?Who Pays the Piper?
Second-generation antipsychotics Second-generation antipsychotics for schizophreniafor schizophrenia
Head-to-head comparisonsHead-to-head comparisons Contradictory evidence…Contradictory evidence…
……depending on who paid for the depending on who paid for the studystudy
Who Pays the Piper?Who Pays the Piper?
Comparison DrugsComparison Drugs Favorable reportsFavorable reports
SponsorSponsor Sponsor’sSponsor’s ComparisonComparison
OlanzapineOlanzapine vs vs risperidonerisperidone
LillyLilly 55 00
JanssenJanssen 33 11
OlanzapineOlanzapine vs vs clozapineclozapine
LillyLilly 22 11
NovartisNovartis 11 00
Who Pays the Piper?Who Pays the Piper?
Comparison DrugsComparison Drugs Favorable reportsFavorable reports
SponsorSponsor Sponsor’sSponsor’s ComparisonComparison
ClozapineClozapine vs vs risperidonerisperidone
NovartisNovartis 11 00
JanssenJanssen 11 00
ZiprasidoneZiprasidone vs vs olanzapineolanzapine
PfizerPfizer 11 11
LillyLilly 22 00
Who Pays the Piper?Who Pays the Piper?
Comparison DrugsComparison Drugs Favorable reportsFavorable reports
SponsorSponsor Sponsor’sSponsor’s ComparisonComparison
Amisulpride vs Amisulpride vs olanzapineolanzapine
LillyLilly 11 00
Sanofi-SyntholabSanofi-Syntholab 11 00
Of 30 studies, 27 (90%) showed the Of 30 studies, 27 (90%) showed the drug made by the company that drug made by the company that sponsored the study was bettersponsored the study was better
McClatchy NewspapersMcClatchy Newspapers2 February 20092 February 2009
““In the waning days of the Bush In the waning days of the Bush administration, the FDA issued new administration, the FDA issued new guidelines to make it easier for drug guidelines to make it easier for drug manufacturers to promote ‘off-label’ manufacturers to promote ‘off-label’ prescription drug use…”prescription drug use…”
“…“…doctors can find good information doctors can find good information on new drug uses in medical on new drug uses in medical journals…”journals…”
Quote of the YearQuote of the Year
““It is simply no longer possible to It is simply no longer possible to believe much of the clinical research believe much of the clinical research that is published, or to rely on the that is published, or to rely on the judgment of trusted physicians or judgment of trusted physicians or authoritative medical guidelines.”authoritative medical guidelines.”
Marcia Angell, MD Marcia Angell, MD Past Editor, New England Journal of Medicine Past Editor, New England Journal of Medicine
New York Review of Books; 15 December 2008New York Review of Books; 15 December 2008
Quote from ‘09Quote from ‘09
““Far too large a section of the treatment Far too large a section of the treatment of disease is today controlled by the big of disease is today controlled by the big manufacturing pharmacists, who have manufacturing pharmacists, who have enslaved us in a plausible pseudo-enslaved us in a plausible pseudo-science...The blind faith which some science...The blind faith which some men have in medicines illustrates too men have in medicines illustrates too often the greatest of all human often the greatest of all human capacities - the capacity for self capacities - the capacity for self deception...”deception...”
William Osler – June 1909William Osler – June 1909
““Far too large a section of the treatment Far too large a section of the treatment of disease is today controlled by the big of disease is today controlled by the big manufacturing pharmacists, who have manufacturing pharmacists, who have enslaved us in a plausible pseudo-enslaved us in a plausible pseudo-science...The blind faith which some science...The blind faith which some men have in medicines illustrates too men have in medicines illustrates too often the greatest of all human often the greatest of all human capacities - the capacity for self capacities - the capacity for self deception...”deception...”
And Finally…And Finally…
When you see the phrase, “This When you see the phrase, “This drug (or test) may be useful…”drug (or test) may be useful…”
……it is an incomplete sentenceit is an incomplete sentence It should say, “This drug (or test) It should say, “This drug (or test)
may may or may notor may not be useful.” be useful.”
Three Takeaway PointsThree Takeaway Points
1.1. About 50% of what you are About 50% of what you are learning is wrong. Nobody learning is wrong. Nobody knows which 50%.knows which 50%.
2.2. 90% of medical literature is crud90% of medical literature is crud““90% of everything is crud.”90% of everything is crud.”
Sturgeon’s Law (1958)Sturgeon’s Law (1958)
Three Takeaway PointsThree Takeaway Points
3.3. Love your teachers as you love Love your teachers as you love your parents, but never forget your parents, but never forget that your teachers are just as full that your teachers are just as full of BS as your parents.of BS as your parents.