The golden standard or fool’s goldSelective data, impact on safety

Clinical trials on trialHAI Europe Open Seminar 2008

Neues Stadthaus, Berlin, 21 November 2008

Joan-Ramon Laporte

RCTs – The golden standard or fool’s gold?

Internal validity

External validity - Efficacy vs Effectiveness

Publication bias

Fraud

Internal validity

• Did the control group receive optimal treatment?

Internal validity

• Did the control group receive optimal treatment?

• Was the dose of the control group adequate?

Favoursatypical

Favourshaloperidol

Drop out rates by dose of comparator drug in trials of patientswith schizophrenia or related disorders (risk difference and95 % confidence intervals)

Geddes et al., 2000

≤ 12 mg haloperidol

> 12 mg haloperidol

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1

Internal validity

• Did the control group receive optimal treatment?

• Was the sample size adequate to identify any relevant difference?

• Was the dose of the control group adequate?

Internal validity

• Did the control group receive optimal treatment?

• Did the published results refer to the primary variable?

• Was the sample size adequate to identify any relevant difference?

• Was the dose of the control group adequate?

Internal validity

• Did the control group receive optimal treatment?

• Have all the trial results been published?

• Did the published results refer to the primary variable?

• Was the sample size adequate to identify any relevant difference?

• Was the dose of the control group adequate?

Internal validity

• Did the control group receive optimal treatment?

• Were the results presented as a relative risk reduction, or as an absolute risk reduction?

• Have all the trial results been published?

• Did the published results refer to the primary variable?

• Was the sample size adequate to identify any relevant difference?

• Was the dose of the control group adequate?

RCTs – The golden standard or fool’s gold?

Internal validity

External validity - efficacy vs effectiveness

Publication bias

Fraud

External validity of clinical trials

• Context

• Reference population

• Selection criteria

• Diagnostic criteria

• Follow up

• Treatments (doses, compliance)

• Duration

• Primary and other variables

• Adverse effects

Lancet 2005; 365: 82-93

(Un)transferability to clinical practice

Patients in RCTs differ from those in real practice:– Age– Comorbidity– Other treatments– Doses taken, compliance– Diagnostic criteria

Efficacy vs effectiveness

RCT UCP

Nº of patients 102-103 104-107

Duration Short Longer

Populations High risk groups Potentially the excluded whole

population Comorbidity Generally excluded Often present

Conditions Well defined Ill-defined

Nº of drugs One or limited Undetermined

Dose/dosage Generally constant Often variable

Pattern of use Continuous Intermittent

Follow up Careful Less careful

RCTs – The golden standard or fool’s gold?

Internal validity

External validity - efficacy vs effectiveness

Publication bias

Fraud

RCTs – The golden standard or fool’s gold?

Internal validity

External validity - efficacy vs effectiveness

Publication bias

Fraud

Third Report of the Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol

in Adults (Adult Treatment Panel III) (ATP III)

Financial Disclosure: Dr Grundy has received honoraria from Merck, Pfizer, Sankyo, Bayer, and Bristol-Myers Squibb. Dr Hunninghake has current grants from Merck, Pfizer, Kos Pharmaceuticals, Schering Plough, Wyeth Ayerst, Sankyo, Bayer, AstraZeneca, Bristol-Myers Squibb, and G. D. Searle; he has also received consulting honoraria from Merck, Pfizer, Kos Pharmaceuticals, Sankyo, AstraZeneca, and Bayer. Dr McBride has received grants and/or research support from Pfizer, Merck, Parke-Davis, and AstraZeneca; has served as a consultant for Kos Pharmaceuticals, Abbott, and Merck; and has received honoraria from Abbott, Bristol-Myers Squibb, Novartis, Merck, Kos Pharmaceuticals, Parke-Davis, Pfizer, and DuPont. Dr Pasternak has served as a consultant for and received honoraria from Merck, Pfizer, and Kos Pharmaceuticals, and has received grants from Merck and Pfizer. Dr Stone has served as a consultant and/or received honoraria for lectures from Abbott, Bayer, Bristol-Myers Squibb, Kos Pharmaceuticals, Merck, Novartis, Parke-Davis/Pfizer, and Sankyo. Dr Schwartz has served as a consultant for and/or conducted research funded by Bristol-Myers Squibb, AstraZeneca, Merck, Johnson & Johnson-Merck, and Pfizer.

Conclusions

• The RCT is the best epidemiological method for causal inference

• However, it is often performed in a way which favours the sponsor’s treatment:– In its design

– In data analysis and interpretation

– In the publication of results

• At best, RCTs are one of many pieces of evidence about therapeutic interventions

Conclusions

• The appraisal of innovation should not only take into account the so-called EBM, but also other evidence:– Pharmacodynamics– Pharmacokinetics– Availability of therapeutic alternatives

• The medical literature is no longer reliable for valid information

• Research should be performed, analyzed and published in a way which should be independent from commercially interested parties