Why we misunderstand medical evidence:A view from the shop floorDR. JAMES GALLOWAYCLINICAL LECTURER AND CONSULTANT RHEUMATOLOGISTKING’S COLLEGE LONDON

Focus of the modern medical school curriculum:

Patient centered care

Emphasis on communication

Safe practice

Medical education Medical students have 5 years to gain a superficial understanding of human health and disease

Post Graduation Learning becomes more disease focused

Attention to practice guidelines◦ Areas of uncertainty (artificially) diminishing

Relevance of methodology understated◦ Clinicians encouraged to just get on and do research◦ Methodological support sparse, sometimes confrontational and often

lacking in enthusiasm

“you should have come and seen me months ago, not 24 hours before your deadline…”

“no, my p value isn’t wrong, you collected too few data. And yes, the word data is plural.”

Designing Clinical Research

Ideas, questions, hypotheses

Idea◦ I have seen quite a few patients on anti-TNF therapy admitted to hospital

with infections

Question◦ Does anti-TNF therapy a predispose people to infections?

Hypothesis◦ Patients on anti-TNF therapy are more likely to experience a serious infection

compared to patients treated with other anti-rheumatic treatments

Designing Clinical Research

A wish list for my trainees:

1) They should understand what a p value means

(and why a series of good anecdotes ≠ evidence)

P values on a pedestal

Renal carcinoma in the US

The counties with the lowest incidence of cancer are mostly rural, sparsely populated and located in traditionally Republican states in the Midwest, the South and the West.

“less pollution, no food additives, healthier lifestyle”

Renal carcinoma in the US

The counties with the highest incidence of cancer are mostly rural, sparsely populated and located in traditionally Republican states in the Midwest, the South and the West…

Training clinicians not to see patterns when they do not exist

What do Google, Facebook and Twitter have in common?

◦ They all have brand names that repeating letters (‘oo’ or ‘tt’)

Heuristics and medicine:

A 21 year old female has been admitted with chest pain. She has a rash.

◦ Emergency physician: “Panic attack”◦ Pulmonologist: “Pulmonary embolism”◦ Cardiologist: “Myocarditis”◦ Rheumatologist: “Lupus”

Designing clinical research

A wish list for my trainees:

2) They should understand the difference between statistical significance and clinical relevance

Interventional studies

Conclusion:

“This study demonstrates that CS improves hand pain and function in patients with symptomatic OA of the hand and shows a good safety profile.”

Gabay et al. Arthritis & Rheumatism, 2011

Designing clinical research

A wish list for my trainees:

3) They should understand the concept of statistical power

A negative result does not mean a drug does not work…

Sample sizes

(amount of money I have)*√(random estimate of how many people might agree to participate)

n =(cost of one patient being put through trial)

Sample sizes It is unethical to under power a study

Sample sizes It is unethical to under power a study

Methotrexate in Giant Cell Arteritis?

◦ Meta analysis of 3 RCTs◦ Hazard ratio for relapse: 0.65 (p = 0.4)

Damascene meta analysis BMJ 2014

Designing clinical research

A wish list for my trainees:

4) They should understand the difference between relative and absolute risks

50% increased cardiovascular risk

Incidence rates (BSRBR data)

Results nbDMARD All TNF

Follow-up, patient-years 9259 36,230

Number of SIs 296 1512

Rate/1000 patient-years (95% CI) 32 (28–36) 42 (40–44)

Unadjusted HR Ref. 1.5 (1.3–1.7)

adjHR* (95% CI) Ref. 1.2 (1.1–1.5)

Follow-up, months 0–6 Ref. 1.8 (1.2–2.6)

6–12 Ref. 1.4 (0.9–2.0)

12–24 Ref. 1.2 (0.8–1.6)

24–36 Ref. 0.9 (0.6–1.3)

Rheumatology (Oxford). 2011;50:124-31

Hazard ratios (BSRBR data)

Results nbDMARD All TNF

Follow-up, patient-years 9259 36,230

Number of SIs 296 1512

Rate/1000 patient-years (95% CI) 32 (28–36) 42 (40–44)

Unadjusted HR Ref. 1.5 (1.3–1.7)

adjHR* (95% CI) Ref. 1.2 (1.1–1.5)

Follow-up, months 0–6 Ref. 1.8 (1.2–2.6)

6–12 Ref. 1.4 (0.9–2.0)

12–24 Ref. 1.2 (0.8–1.6)

24–36 Ref. 0.9 (0.6–1.3)

Rheumatology (Oxford). 2011;50:124-31

Describing risk to patients

Absolute

“On anti-TNF therapy, your risk of infection is increased by 1%”

Relative

“On anti-TNF therapy, your risk of infection is increased by 50%”

Frustratingly, both of these statements are technically correct

• The infection rate on anti-TNF is 4% per year, which is 1 percentage point higher than that observed in the comparator group (3%)

• However, relatively speaking, the rate of infection in people on anti-TNF increases by 50%

Designing clinical research

A wish list for my trainees:

5) They should understand causal inference

Causal / casual assumptions

“Methotrexate is an important cause of chronic lung damage in patients with rheumatoid arthritis”

Channeling biasDisease

OutcomeIntervention

Causal inference

In search of causality

Making sense of it allYes, take a closer look.

You can’t be serious?

Well, just to be safe I’m going to

assume that cancer causes mobile

phones.

This association between mobile phone use and

cancer is fascinating isn’t it?

Well, I actually think they have it all the wrong way round.

Designing clinical research

A wish list for my trainees:

6) They should understand the sensitivity and specificity

Quick test A 50 year old woman undergoes screening mammography. Her mammogram is positive.

She is worried and asks her doctor what the chance of her having breast cancer is based upon this test result.

What is the probability that she has breast cancer?OPTIONS

a) 9 in 10

b) 8 in 10

c) 1 in 10

d) 1 in 100

BACKGROUND INFORMATION

The probability that a woman has breast cancer is 1% ("prevalence")

If a woman has breast cancer, the probability that she tests positive is 90% ("sensitivity")

If a woman does not have breast cancer, the probability that she nevertheless tests positive is 9% ("false alarm rate")

Results Out of 160 respondents (all doctors) only 21% answered correctly.

◦ Most thought the correct answer was 9 in 10.

1000 women

10 have cancer

9 test +ve

1 tests -ve

990 don’t

89 test +ve

901 test -ve

1 in 10

My methodology wish list for doctors:

1) Understand why statistics are needed

2) Appreciate significance ≠ relevance

3) Recognise the value of statistical power

4) Comprehend absolute and relative risk

5) Understand concept of causation

6) Distinguish sensitivity and specificity

Teaching the teachers Challenge trainees

Use active learning (keep it relevant)

Ensure teachers care about the subject

Teachers must be credible leaders

Feedback, feedback, feedback

◦ Move with the times… Facebook, Twitter…

In conclusion Doctors have a skill set that does not naturally align with mathematics

The emphasis of the modern curriculum has evolved to a less scientific model

◦ The result of a weak understanding of statistics is that doctors misinterpret evidence and patients are misinformed about risk and benefit

We need to make medical statistics more prominent in the curriculum

Final wish… All students should be taught to mistrust pie charts and anyone who uses them.