Are you at risk?Are you at risk?

falling victim to misleading falling victim to misleading presentations as a result of presentations as a result of conflict of interest of conflict of interest of investigatorsinvestigators

Not at allNot at all A littleA little Moderate riskModerate risk High riskHigh risk

SpinningSpinning

• get in groups of 2 to 5get in groups of 2 to 5

• write down three ways to avoid write down three ways to avoid being misleadbeing mislead

• 3 minutes3 minutes

Conclusions and Funding, Conclusions and Funding, Als-Nielsen, JAMA 2003Als-Nielsen, JAMA 2003

• 25 Cochrane reviews, 370 RCTs25 Cochrane reviews, 370 RCTs– random selection of 167 of 1081 (issue 2, 2001)random selection of 167 of 1081 (issue 2, 2001)– excluded excluded

• 101 < 5 studies 101 < 5 studies • 6 no concealment variability 6 no concealment variability • 2 no binary outcome 2 no binary outcome • 16 non-pharmacological 16 non-pharmacological

• abstractedabstracted– sources of fundingsources of funding– effect on primary outcomeeffect on primary outcome– adverse effectsadverse effects– methodological quality (concealment, blinding)methodological quality (concealment, blinding)

Scale Used to Grade Conclusions in Scale Used to Grade Conclusions in TrialsTrials

6 Points6 PointsExperimental intervention highly preferred and should now be Experimental intervention highly preferred and should now be

considered the standard intervention in all patients, or similarconsidered the standard intervention in all patients, or similar5 Points5 PointsExperimental intervention preferred to control, but further trials still Experimental intervention preferred to control, but further trials still

indicated, experimental may be more costly, or similarindicated, experimental may be more costly, or similar4 Points4 PointsExperimental and control intervention about equal, but the Experimental and control intervention about equal, but the

experimental cheaper, easier to administer, or similar minor experimental cheaper, easier to administer, or similar minor advantageadvantage

3 Points3 PointsExperimental and control intervention about equal, but the control Experimental and control intervention about equal, but the control

may be cheaper, easier to administer, or similar minor advantagemay be cheaper, easier to administer, or similar minor advantage2 Points2 PointsControl intervention preferred to experimental intervention, but Control intervention preferred to experimental intervention, but

experimental intervention might be promising under some experimental intervention might be promising under some circumstances, or similarcircumstances, or similar

1 Point1 PointControl intervention highly preferred and should now be considered Control intervention highly preferred and should now be considered

the standard intervention in all patients, or similarthe standard intervention in all patients, or similar

Relation Between Funding Source and Relation Between Funding Source and Conclusions in 370 Randomized Drug Conclusions in 370 Randomized Drug

TrialsTrials

Abbreviation: IQR, interquartile range.*Conclusions in trials were assessed by a 1-6 point scale. If the conclusions recommended the experimental drug as the treatment of choice without disclaimers, 6 points was assigned, and if not, 1-5 points was assigned.+p<.001, using Kruskal-Wallis test (medians) or X2 test (proportions)

FundingFunding # of # of TrialsTrials

Median Median ScoreScore

(IQR)+(IQR)+

# (%) of Trials # (%) of Trials Scoring 6 Scoring 6 Points+Points+

Nonprofit Nonprofit organizationsorganizations

6767 4 (3-5)4 (3-5) 11 (16.4)11 (16.4)

Not reportedNot reported 106106 5 (3-6)5 (3-6) 32 (30.1)32 (30.1)

Nonprofit and for-Nonprofit and for-profit organizationsprofit organizations

5151 5 (4-6)5 (4-6) 18 (35.2)18 (35.2)

For-profit For-profit organizationsorganizations

146146 6 (5-6)6 (5-6) 74 (50.6)74 (50.6)

TotalTotal 370370 5 (4-6)5 (4-6) 135 (36.4)135 (36.4)

Als-Nielson, JAMA, 2003Als-Nielson, JAMA, 2003

• bigger effect size, more likely bigger effect size, more likely stronger recommendationstronger recommendation

• blinding, more likely stronger blinding, more likely stronger recommendationrecommendation

• after adjustment, industry funding, after adjustment, industry funding, odds ratio 5.3 (95% CI 2.0 to 14.4)odds ratio 5.3 (95% CI 2.0 to 14.4)

Read methods and resultsRead methods and resultsbypass the discussionbypass the discussion

1

Use of placebo when active comparator optimal

Angiotensin receptor blockers for diabetic nephropathy

vs.

ACE inhibitors

Parving H-H et al. N Engl J Med 2001;345(12):870-878Brenner BM N Engl J Med 2001;345(12):861-869Lewis EJ et al. N Engl J Med 2001;345(12):851-60.

Hostetter TH. N Engl J Med 2001;345(12):910-912

Do you manage patients with type II diabetes?

Is there any agent you would routinely suggest forpatients with diabetic nephropathy?

Incomparable comparatorsIncomparable comparators

• 8 RCTS of 28 RCTS of 2ndnd generation generation neuroleptics vs. 20 mg/d haloperidolneuroleptics vs. 20 mg/d haloperidol– fewer extrapyramidal symptomsfewer extrapyramidal symptoms– standard dose haloperidol < 12 mgstandard dose haloperidol < 12 mg

• RCT paroxetine qd vs. amitriptyline RCT paroxetine qd vs. amitriptyline bidbid– less daytime somnolenceless daytime somnolence– standard amitriptyline qhsstandard amitriptyline qhsSafer J Nerv Ment Dis 2002;190(9):583-92.

Geddes J et al BMJ 2000;321(7273):1371-6.Christiansen PE, et al. Acta Psychiatr Scand 1996;93(3):158-63

Beware faulty Beware faulty comparators comparators

2

Irbesartan vs amlodipine Irbesartan vs amlodipine in diabetic nephropathyin diabetic nephropathy

• in comparison to amlodipine, irbesartan in comparison to amlodipine, irbesartan reduced the combined endpoint of all reduced the combined endpoint of all cause mortality, progression to end stage cause mortality, progression to end stage renal disease, and doubling of serum renal disease, and doubling of serum creatinine (RRR 20%, 95% CI 7.5% to creatinine (RRR 20%, 95% CI 7.5% to 32%)32%)

• did irbesartan reduce all-cause mortality?did irbesartan reduce all-cause mortality?

-40 -24 -8 8 24 40 56

Risk reduction with irbesartan (vs. amlodipine)

RRR (95% CI)

Composite endpoint

All-cause mortality

End-stage renal disease

Doubling of creatinine concentration

RCT 1715 DM 2 nephropathy, HTN irbesartan vs amlodipine NEJM 2001; 345: 851

RRR 20% (7.5-32%)

RRR 33% (16-47%)

RRR 23% (-0.5-41%)

RRR -3% (-35-22%)

What has gone wrong?What has gone wrong?

• widely varying importancewidely varying importance

• biggest effect on least importantbiggest effect on least important– most important no effectmost important no effect

• criteria for use of compositecriteria for use of composite– similar patient-importancesimilar patient-importance– similar effectsimilar effect

Beware composite Beware composite endpointsendpoints

3

5,269 patients with abnormal glucose tolerance test

randomized to lifestyle advice, or advice + rosiglitazone

followed for 3 to 5 years

What is the authors’ message?

• rosiglitazone to prevent diabetes:

• strong indication (for all)

• weak indication (for some)

• not indicated

Doctor, what do I gain by Doctor, what do I gain by taking rosiglitazone?taking rosiglitazone?

• Doc: less chance of diabetesDoc: less chance of diabetes

• Pt: what happens if I get diabetesPt: what happens if I get diabetes

• Doc: you have to take a drugDoc: you have to take a drug

• Pt: the same drug I’m taking to prevent diabetes?Pt: the same drug I’m taking to prevent diabetes?

• Doc: I could give you a drug with less problemsDoc: I could give you a drug with less problems

• Pt: I’ll take a drug every day for 3 years to lower my Pt: I’ll take a drug every day for 3 years to lower my risk of taking the same or a less toxic drug from 25 to risk of taking the same or a less toxic drug from 25 to 10%???10%???

drug no drug difference

diabetes medication 3 years 30000 3650 26350

new diagnosis of diabetes 1060 2500HR 0.38 (CI, 0.33-

0.44)

Anxiety about diabetes ?? ?? ??

Costs inconvenience self-monitoring ?? ?? ??

Costs and inconvenience HbA1c, lipoprotein testing, retinal exam, etc.

?? ?? ??

MI, stroke, CV death at 3 years 120 90HR 1.39 (CI 0.81-

2.37)

Heart failure, at 3 years 50 10 HR 7.03 (CI 1.6-30.9)

kidney, eyes, neuropathy ?? ?? ??

Peripheral edema, at 3 years 680 490 RR 1.4 (CI 1.1-1.8)

Weight gain (kg), at 3 years + 1.1 -1.1 + 2.2

Rare (fractures, macular edema) ?? ?? ??

What is your view?

• rosiglitazone to prevent diabetes:

• strong indication (for all)

• weak indication (for some)

• not indicated

• what has gone wrong here?

Other problematic surrogatesOther problematic surrogates• tests of cognitive function instead of function and tests of cognitive function instead of function and

behavior in Alzheimer’sbehavior in Alzheimer’s

• bone density instead of fractures in osteoporosisbone density instead of fractures in osteoporosis

• oxygenation instead of mortality in ARDSoxygenation instead of mortality in ARDS

• asymptomatic DVT instead of symptomatic asymptomatic DVT instead of symptomatic thromboembolism thromboembolism

• pulmonary function instead of exacerbations and pulmonary function instead of exacerbations and qol in respiratory diseaseqol in respiratory disease

• lipids instead of CV events in atherosclerosislipids instead of CV events in atherosclerosis

Beware substitute Beware substitute endpointsendpoints

4

Five vs Four Courses of Five vs Four Courses of Therapy for Acute Myeloid Therapy for Acute Myeloid

LeukemiaLeukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66-70

Five vs Four Courses of Five vs Four Courses of Therapy for Acute Myeloid Therapy for Acute Myeloid

LeukemiaLeukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66-70

Five vs Four Courses of Five vs Four Courses of Therapy for Acute Myeloid Therapy for Acute Myeloid

LeukemiaLeukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66-70

Five vs Four Courses of Five vs Four Courses of Therapy for Acute Myeloid Therapy for Acute Myeloid

LeukemiaLeukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66-70

True beneficial effect

No effect

Stopping boundary

True beneficial effect

No effect

Stopping boundary

Look after every patient or event

stop

True beneficial effect

No effect

Stopping boundary

Interim analyses every q patients or events

stopstop

stop

Beware trials stopped Beware trials stopped early early

5

Users Guides to SpinUsers Guides to Spin

1.1. read methods and results; bypass read methods and results; bypass the discussion sectionthe discussion section

2.2. beware faulty comparatorsbeware faulty comparators

3.3. beware composite endpointsbeware composite endpoints

4.4. beware of substitute endpointsbeware of substitute endpoints

5.5. beware trials stopped earlybeware trials stopped early

Wall St. Journal, November, 2006

Wall St. Journal, November, 2006

New York Times, May 9, 2007

Available at: www.mhprofessional.com/jama