EBMRC

How to Use an Article About a Diagnostic Test

Akbar Soltani. MD, EndocrinologistTehran University of Medical Sciences (TUMS)

Endocrine and Metabolism Research Center (EMRC)Evidence-Based Medicine Research Center (EBMRC)

Shariati Hospitalwww.soltaniebm.com

www.ebm.irwww.avincennact.ir

EBMRC

OBJECTIVES•Objectives of testing•Diagnostic research•Critical appraisal•Summary

EBMRC

• 2/3 legal claims against GPs in UK

• 40,000-80,000 US hospital deaths from misdiagnosis per year

• Adverse events, negligence cases, serious disability more likely to be related to misdiagnosis than drug errors

• Diagnosis uses <5% of hospital costs, but influences 60% of decision making

EBMRC

Objectives of testing • Increasing certainty of the

presence or absence of disease • This requires sufficient

discriminative power. • 2×2 table relating test outcome

to a reference standard.

EBMRC

Objectives of testing

• Supporting clinical management

• Determining localization, and shape of arterial lesions is necessary for treatment decisions

EBMRC

Objectives of testing

• Assessing prognosis As the starting point for clinical follow up and informing patients.

• Monitoring clinical course When a disease is untreated, or during or after treatment.

• Measuring fitness For example, for sporting activity or for employment.

EBMRC

OBJECTIVES•Objectives of testing•Diagnostic research•Critical appraisal•Summary

EBMRC

Evidence Based Approach

How to Use an Article About a Diagnostic Test

EBMRC

Appraising diagnostic tests: 3 easy steps

1. Are the results valid?

2. What are the results?

3. How Can the Results be Applied to Patient Care?

EBMRC

Some definitions

True positive

A

False positive

B

False negative

C

True negative

D

Disease Present Absent

Positive

Negative

TestResult

Sensitivity = A / (A+C)

Specificity = D / (B+D)

EBMRC

Tip…..

• Sensitivity is useful to me– ‘The new rapid chlamydia test was positive in 47 out of 56 women with

chlamydia (sensitivity =83.9%)’

• Specificity seems a bit confusing!– ‘The new rapid chlamydia test was negative in 600 of the 607 women

who did not have chlamydia (specificity = 98.8%)’

• So…the false positive rate is sometimes easier– False positive rate = 1 – specificity– So a specificity of 98.8% means that the new rapid test is wrong (or

falsely positive) in 1.2% of women

EBMRC

Some definitions

True positive

A

False positive

B

False negative

C

True negative

D

Disease Present Absent

Positive

Negative

TestResult

PPV = A / (A+B)

NPV = D / (C+D)

Sen*prevalence Sen*prev+(1-Spec) *(1-prev)

Spec*(1-prev)(1-Sen)*prev+ Spec*(1-prev)

EBMRC

Series of patientsSeries of patients

Index testIndex test

Reference (“gold”) standardReference (“gold”) standard

Compare the results of the Compare the results of the index test with the reference index test with the reference

standard, blindedstandard, blinded

Basic structure of diagnostic studiesBasic structure of diagnostic studies

EBMRC

Appraising diagnostic tests: 3 easy steps

1. Are the results valid?

2. What are the results?

3. How Can the Results be Applied to Patient Care?

EBMRC

• What were the key selection (inclusion & exclusion) criteria?

• Were they replicable?

• List important selection criteria; e.g. age group, gender, risk profile, medical history.

• There should be sufficient information in the paper to allow the reader to theoretically select a similar population

EBMRC

• Did selection lead to an appropriate spectrum of participants (like those assessed in practice)

• Participants with the range of common presentations of the target disorder and with commonly confused diagnosis

EBMRC

1. Was an appropriate spectrum of patients included? Spectrum bias

• You want to find out how good chest X rays are for diagnosing pneumonia in the Emergency Department

• Best = all patients presenting with difficulty breathing get a chest X-ray

• Spectrum bias = only those patients in whom you really suspect pneumonia get a chest X ray

EBMRC

Spectrum bias

• Studies from referral centers

• Patients with negative results are less likely to referred

EBMRC

2) Did the results of the test being evaluated influence the decision to perform the reference standard? 2. Were all patients

subjected to the gold standard? Verification (work-up) bias

• What was the gold standard of diagnosis?

• The validity of the study requires that there is an accepted, valid and replicable reference standard of diagnosis.

EBMRC

gold standard• 1.Laboratory tests (Infectious & endocrine diseases)• 2.Imaging (DVT, PTE )• 3.Biopsy

(Cancer , vasculitis)• 4.Autopsy (neurologic diseases)• 5.long-term follow-up (SLE , MS)

EBMRC

2) Did the results of the test being evaluated influence the decision to perform the reference standard? 2. Were all patients

subjected to the gold standard? Verification (work-up) bias

• You want to find out how good is exercise ECG (“treadmill test”) for identifying patients with angina

• The gold standard is angiography

• Best = all patients get angiography

• Verification (work-up bias) = only patients who have a positive exercise ECG get angiography

EBMRC

• You want to find out how good is exercise ECG (“treadmill test”) for identifying patients with angina

• All patients get the gold standard (angiography)• Observer bias = the Cardiologist who does the

angiography knows what the exercise ECG showed (not blinded)

• Another example: The pulmonary nodule on CT, and comparison to CXR

3. Was there an independent, blind or objective comparison with the gold standard? Observer bias

EBMRC

4) Were the methods for performing the test described in sufficient detail to permit replication?

• This description should cover all issues that are important in the preparation of the patient (diet, drugs to be avoided, precautions after the test), the performance of the test (technique, possibility of pain), and the analysis and interpretation of its results.

EBMRC

Series of patientsSeries of patients

Index testIndex test

Reference (“gold”) standardReference (“gold”) standard

Compare the results of the Compare the results of the index test with the reference index test with the reference

standard, blindedstandard, blinded

Basic structure of diagnostic studiesBasic structure of diagnostic studies

EBMRC

Read this abstract

• Scan in UTI abstract

EBMRC

• Scan in UTI abstract

Index test

Series of patients

Gold standard

Accuracy

EBMRC

Series of patients

Index test

Reference (“gold”) standard

Compare the results of the index test with the reference

standard

Women presenting with history of recurrent UTIs

Self diagnosis based on symptoms

Positive urine culture

172 episodes:

Positive urine culture in 144 (84%)

EBMRC

Do reports meet the standards?

• Between 1978 and 1993 the authors found 112 articles, predominantly in radiological tests and immunoassays.

• Few of the standards were met consistently - ranging from 46% avoiding workup bias down to 9% reporting accuracy in subgroups.

EBMRC

EBMRC

EBMRC

EBMRC

Appraising diagnostic tests: 3 easy steps

1. Are the results valid?

2. What are the results?

3. How Can the Results be Applied to Patient Care?

EBMRC

What are the Results?Biostatistics Review

• Sensitivity : Of all the people with a particular disease, the proportion who will test positive for it (PID)

• Specificity : Of all the people without the disease, the proportion who will test negative for it (NIH)

A gold standard diagnosis is already known, (presumably without error). The above terms are describing the properties of a particular TEST.

EBMRC

What are the Results?Biostatistics Review

Example: Suppose the Prevalence of a particular disease in a population is 10%. (of 1000 people, 100 have the disease)

Sensitivity: 90/100 = 90%

Specificity: 810/900 = 90%Dz (+) Dz (-)

Test(+) 90 90Test(-) 10 810

100 900

EBMRC

What are the Results?Biostatistics Review

• Positive Predictive Value : Of all the people who tested positive for a disease, the proportion that actually has it

• Negative Predictive Value : Of all the people who tested negative for a disease, the proportion that actually does not have it

In these patients, what you know are their test results, from which you are trying to determine whether they actually have the disease.

EBMRC

What are the Results?Biostatistics Review

Same example: Prevalence of a particular disease in the population is 10%.

PPV: 90/180 = 50%

NPV: 810/820 = 99%

Dz (+) Dz (-)

Test(+) 90 90 180Test(-) 10 810 820 Sens: 540/600 = 90%

Spec: 360/400 = 90%

EBMRC

What are the Results?Note: The PPV and NPV of a test are DEPENDENT on

prevalence.

Suppose the Disease Prevalence is 60%

Sens: 540/600 = 90%

Spec: 360/400 = 90%

PPV: 540/580 = 93%

NPV: 360/420 = 86%

Dz (+) Dz (-)

Test(+) 540 40 580Test(-) 60 360 420

600 400

EBMRC

What are the Results?The use of PPV and NPV when describing a diagnostic test

has several drawbacks:

1. They are dependent on the prevalence of disease2. The prevalence of disease in the general population is

not the same as that of the patients you see in clinic/ER.3. Not all tests have results that can be categorized as “+”

or “-”.

For these reasons, PPV and NPV may soon be considered “Old School”.

EBMRC

In My Experience!

Likelihood of disease and the PPVLikelihood of disease and the PPVof diagnostic tests in specialty settingof diagnostic tests in specialty setting

A test with 98% sen &spA test with 98% sen &spPrimary care setting p=2% ppv=50%Primary care setting p=2% ppv=50%Specialty setting p=30% ppv=95%Specialty setting p=30% ppv=95%

EBMRC

A disease with a prevalence of 4% must be diagnosed.

It has a sensitivity of 50% and a specificity of 90%.

If the patient tests positive, what is the chance they have the disease?

Try itTry it

EBMRC

Prevalence of 4%, Sensitivity of 50%, Specificity of 90%

4

96

2

9.6

100

11.6 people test positive……….

of whom 2 have the disease

So, chance of disease is 2/11.6 about 17%

Disease +ve

Disease -ve

Testing +ve

Sensitivity = 50%

False positive rate = 10%

EBMRC

Prevalence of 4%, Sensitivity of 50%, Specificity of 90%

• Doctors with an average of 14 yrs experience

….answers ranged from 1% to 99%

….half of them estimating the probability as 50%

Gigerenzer G BMJ 2003;327:741-744

EBMRC

What are the Results?

Likelihood Ratios

Simply stated, the Likelihood Ratio is how much more likely someone is to get a positive test result if they have the disease, compared to someone who doesn’t.

In fact, the LR is how much more likely someone is to get any particular test result (positive, negative, “intermediate probability, etc) if they have disease, compared to someone who doesn’t.

EBMRC

What are the Results?From our previous example:The LR is (90/100) / (90/900) = 9.0

This is the LR for a (+) result. Someone with disease is 9 times as likely to test positive than someone without it.

Note also that :LR = SENS / 1 - SPEC

WOWO

Dz (+) Dz (-)

Test(+) 90 90Test(-) 10 810

100 900

EBMRC

What are the Results?The LR for a negative test is:

(10/100) / (810/900) = 0.11

Someone with disease is 0.11 times as likely (1/9) to test negative than someone without it.

Dz (+) Dz (-)

Test(+) 90 90Test(-) 10 810

100 900

EBMRC

Likelihood RatiosLR = 18High

Mr. A Pre-Test Prob.

15%

Post-Test Prob. 20%

Post-Test ProbabilityPre-Test

Probability

Mrs. BPre-Test Prob.

40%

Post-Test Prob. 90%

LR = 1.2V/Q ScanResults

EBMRC

Application: Using Post-test Probability

Diseaseruled IN

Disease ruled OUT

Disease not

ruled in or out

Above this point,treat

Below this point,no further testing

Determined by:Complications of untreated disease

Risks of therapyComplications of tests

Cost

EBMRC

What do likelihood ratios mean?

LR>10 = strong positive test result

LR<0.1 = strong negative test result

LR=1

No diagnostic value

EBMRC

What do likelihood ratios mean?

McGee: Evidence based Physical Diagnosis (Saunders Elsevier)

EBMRC

? Appendicitis:

McBurney tenderness LR+ = 3.4

Pre test 5%

%

%

Bayesian reasoning

Fagan nomogram

Post test 20%

EBMRC

Appraising diagnostic tests: 3 easy steps

1. Are the results valid?

2. What are the results?

3. How Can the Results be Applied to Patient Care?

EBMRC

How Can the Results be Applied to Patient Care?

Will the results change my management strategy?

• Always ask yourself the above before ordering a test• What is your treatment threshold?• What is your threshold for NOT treating?

If you find that most patients are having test results with a LR near 1.0, the test is not very useful.

EBMRC

How Can the Results be Applied to Patient Care?

Will patients be better off as a result of the test?

Do the benefits of performing the test outweigh the associated risk?

If the answer to this is unclear, should a randomized controlled trial be done to determine if the test actually gives any benefit?

EBMRC

Summary •Objectives of testing•Diagnostic research•Critical appraisal

EBMRC

•THANK YOUTHANK YOU