Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results...
-
Upload
madison-stevenson -
Category
Documents
-
view
218 -
download
2
Transcript of Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results...
![Page 1: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/1.jpg)
Assessing Information from Multilevel (Ordinal) Tests
ROC curves and
Likelihood Ratios for results other than “+” or “-”
Michael A. Kohn, MD, MPP
10/4/2007
![Page 2: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/2.jpg)
Four Main Points
1) Dichotomizing a multi-level test by choosing a fixed cutpoint reduces the value of the test.
2) The ROC curve summarizes the ability of the test to differentiate between D+ and D- individuals.
3) LR(result) = P(result|D+)/P(result|D-) = slope of ROC curve.
4) Pre-Test Odds x LR(result) = Post-Test Odds
![Page 3: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/3.jpg)
Many Tests Are Not Dichotomous
Ordinal
• “-”, “+”, “++”, “+++” for leukocyte esterase on urine dip stick
• “Normal”, “Low Prob”, “Intermediate Prob”, “High Prob” on VQ scan
Continuous
• Systolic Blood Pressure
• WBC Count
![Page 4: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/4.jpg)
Evaluating the Test--Test Characteristics
• For dichotomous tests, we discussed sensitivity P(+|D+) and specificity P(-|D-)
• For multi-level and continuous tests, we will discuss the Receiver Operating Characteristic (ROC) curve
![Page 5: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/5.jpg)
Using the Test Result to Make Decisions about a Patient
• For dichotomous tests, we use the LR(+) if the test is positive and the LR(-) if the test is negative
• For multilevel and continuous tests, we use the LR(r), where r is the result of the test
![Page 6: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/6.jpg)
Septic ArthritisBacterial infection in a joint.
![Page 7: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/7.jpg)
Clinical ScenarioDoes this Adult Patient Have Septic Arthritis?
![Page 8: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/8.jpg)
Clinical ScenarioDoes this Adult Patient Have Septic Arthritis?
A 48-year-old woman with a history of rheumatoid arthritis who has been treated with long-term, low-dose prednisone presents to the emergency department with a 2-day history of a red, swollen right knee that is painful to touch. She reports no prior knee swelling and no recent trauma or knee surgery, illegal drug use, rash, uveitis, or risky sexual behavior. On examination, she is afebrile and has a right knee effusion. Her peripheral white blood cell (WBC) count is 11 000/µL and her erythrocyte sedimentation rate (ESR) is 55 mm/h. An arthrocentesis is performed, and the initial Gram stain is negative.
Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88.
You have the synovial white blood cell (WBC) count.
![Page 9: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/9.jpg)
Clinical ScenarioDoes this Adult Patient Have Septic Arthritis?
Assume pre-test probability of septic arthritis is 0.38.
How do you use the synovial WBC result to determine the likelihood of septic arthritis?
Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88.
![Page 10: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/10.jpg)
Why Not Make It a Dichotomous Test?
Synovial Septic ArthritisWBC Count Yes No
>25,000 77% 27%
≤ 25,000 23% 73%
TOTAL* 100% 100%
*Note that these could have come from a study where the patients with septic arthritis (D+ patients) were sampled separately from those without (D- patients).
Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88.
![Page 11: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/11.jpg)
Why Not Make It a Dichotomous Test?
Sensitivity = 77%Specificity = 73%
LR(+) = 0.77/(1 - 0.73) = 2.9LR(-) = (1 - 0.77)/0.73 = 0.32
“+” = > 25,000/uL “-” = ≤ 25,000/uL
![Page 12: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/12.jpg)
Clinical ScenarioSynovial WBC = 48,000/mL
(Demonstrate LR Slide Rule?)
(Demonstrate Excel?)
Pre-test prob: 0.38
LR(+) = 2.9
Post-Test prob = ?
![Page 13: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/13.jpg)
Clinical ScenarioSynovial WBC = 48,000/mL
Pre-test prob: 0.38
Pre-test odds: 0.38/0.62 = 0.61
LR(+) = 2.9
Post-Test Odds = Pre-Test Odds x LR(+)
= 0.61 x 2.9 = 1.75
Post-Test prob = 1.75/(1.75+1) = 0.64
![Page 14: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/14.jpg)
Clinical ScenarioSynovial WBC = 128,000/mL
Pre-test prob: 0.38
LR(+) = ?
Post-Test prob =?
![Page 15: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/15.jpg)
Clinical Scenario Synovial WBC = 128,000/mL
Pre-test prob: 0.38
Pre-test odds: 0.38/0.62 = 0.61
LR(+) = 2.9 (same as for WBC=48,000!)
Post-Test Odds = Pre-Test Odds x LR(+)
= 0.61 x 2.9 = 1.75
Post-Test prob = 1.75/(1.75+1) = .64
![Page 16: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/16.jpg)
Why Not Make It a Dichotomous Test?
Because you lose information. The risk associated with a synovial WBC=48,000 is equated with the risk associated with WBC=128,000.
Choosing a fixed cutpoint to dichotomize a multi-level or continuous test throws away information and reduces the value of the test.
![Page 17: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/17.jpg)
Main Point 1: Avoid Making Multilevel Tests Dichotomous
Dichotomizing a multi-level or continuous test by choosing a fixed cutpoint reduces the value of the test
![Page 18: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/18.jpg)
WBC (/uL) Interval
% of Septic Arthritis
% of No Septic Arthritis
>100,000 29% 1%
>50,000-100,000 33% 7%
>25,000-50,000 15% 19%
0 - 25,000 23% 73%
TOTAL 100% 100%
![Page 19: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/19.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
0 - 25,000 >25,000-50,000
>50,000-100,000
>100,000
No Septic Arthritis
Septic Arthritis
Synovial Fluid WBC Count
![Page 20: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/20.jpg)
Histogram• Does not reflect prevalence of D+ (Dark
D+ columns add to 100%, Open D- columns add to 100%)
• Sensitivity and specificity depend on the cutpoint chosen to separate “positives” from “negatives”
• The ROC curve is drawn by serially lowering the cutpoint from highest (most abnormal) to lowest (least abnormal).*
* Just said that choosing a fixed cutpoint reduces the value of the test. The key issues are 1) the ROC curve is for evaluating the test, not the patient, and 2) drawing the ROC curve requires varying the cutpoint, not choosing a fixed cutpoint.
![Page 21: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/21.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
0 - 25,000 >25,000-50,000
>50,000-100,000
>100,000
Negative Positive
Cutoff = ∞Sensitivity = 0%1 - Specificity = 0%
![Page 22: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/22.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
0 - 25,000 >25,000-50,000
>50,000-100,000
>100,000
Negative Positive
Cutoff = 100,000Sensitivity = 29%1 - Specificity = 1%
![Page 23: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/23.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
0 - 25,000 >25,000-50,000
>50,000-100,000
>100,000
Negative Positive
Cutoff = 50,000Sensitivity = 62%1 - Specificity = 8%
![Page 24: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/24.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
0 - 25,000 >25,000-50,000
>50,000-100,000
>100,000
Negative Positive
Cutoff = 25,000Sensitivity = 77%1 - Specificity = 27%
![Page 25: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/25.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
0 - 25,000 >25,000-50,000
>50,000-100,000
>100,000
Negative Positive
Cutoff = 0Sensitivity = 100%1 - Specificity = 100%
![Page 26: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/26.jpg)
WBC Count (x1000/uL)
Sensitivity 1 - Specificity
> ∞ 0% 0%
> 100 29% 1%
> 50 62% 8%
> 25 77% 27%
≥ 0 100% 100%
Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88.
![Page 27: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/27.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1 - Specificity
Sen
sitiv
ity
Cutoff > ∞
Cutoff > 100k
Cutoff > 50k
Cutoff > 25k
Cutoff ≥ 0
Area Under Curve = 0.8114
![Page 28: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/28.jpg)
0 5 10 15 20 25 30
WBC Count (1000/uL)
D-: No BacteremiaD+: Bacteremia
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
![Page 29: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/29.jpg)
0 5 10 15 20 25 30
WBC Count (1000/uL)
D-: No BacteremiaD+: Bacteremia
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
![Page 30: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/30.jpg)
0 5 10 15 20 25 30
WBC Count (1000/uL)
D-: No BacteremiaD+: Bacteremia
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
![Page 31: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/31.jpg)
0 5 10 15 20 25 30
WBC Count (1000/uL)
D-: No BacteremiaD+: Bacteremia
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
![Page 32: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/32.jpg)
Test Discriminates Well Between D+ and D-
-40 -20 0 20 40 60Test Result
D-D+
![Page 33: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/33.jpg)
Test Discriminates Well Between D+ and D-
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
1 - Specificity
Se
ns
itiv
ity
![Page 34: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/34.jpg)
Test Discriminates Poorly Between D+ and D-
-40 -20 0 20 40 60Test Result
D-D+
![Page 35: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/35.jpg)
Test Discriminates Poorly Between D+ and D-
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
1 - Specificity
Se
ns
itiv
ity
![Page 36: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/36.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1 - Specificity
Sen
sitiv
ity
Cutoff > ∞
Cutoff > 100k
Cutoff > 50k
Cutoff > 25k
Cutoff ≥ 0
Area Under Curve = 0.8114
Area Under ROC Curve
![Page 37: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/37.jpg)
Area Under ROC Curve
Summary measure of test’s discriminatory ability
Probability that a randomly chosen D+ individual will have a more positive test result than a randomly chosen D- individual
![Page 38: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/38.jpg)
Area Under ROC Curve
• Corresponds to the Mann-Whitney (Wilcoxan Rank Sum) Test Statistic, which is the non-parametric equivalent of Student’s t test.
• Also corresponds to the “c statistic” reported in logistic regression models
![Page 39: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/39.jpg)
Area under ROC curve = 0.8405S
en
sitiv
ity
1 - Specificity0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
![Page 40: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/40.jpg)
“Walking Man” Approach to ROC Curves
• Divide vertical axis into d steps, where d is the number of D+ individuals
• Divide horizontal axis into n steps, where n is the number of D- individuals
• Sort individuals from most to least abnormal test result
• Moving from the first individual (with the most abnormal test result) to the last (with the least abnormal test result)…
![Page 41: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/41.jpg)
“Walking Man” (continued)
• …call out “D” if the individual is D+ and “N” if the individual is D-
• Let the walking man know when you reach a new value of the test
• The walking man takes a step up every time he hears “D” and a step to the right every time he hears “N”
• When you reach a new value of the test, he drops a stone.
![Page 42: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/42.jpg)
Synovial WBC Count in 5 Patients with Septic Arthritis
PatientWBC Count(x 1000/uL)
D1 128
D2 92
D3 64
D4 37
D5 24
![Page 43: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/43.jpg)
Synovial WBC Count in 10 Patient Without Septic ArthritisPatient WBC Count (x 1000)
N1 71
N2 48
N3 37
N4 23
N5 12
N6 12
N7 8
N8 7
N9 6
N10 0
![Page 44: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/44.jpg)
Septic Arthritis No Septic Arthritis
128
92
71
64
48
37 37
24
23
12
12
8
7
6
0
![Page 45: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/45.jpg)
DDNDN(DN)DN(NN)NNNN
![Page 46: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/46.jpg)
0
1
2
3
4
5
0 1 2 3 4 5 6 7 8 9 10
D-
D+
![Page 47: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/47.jpg)
0
1
2
3
4
5
0 1 2 3 4 5 6 7 8 9 10
D-
D+
![Page 48: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/48.jpg)
Area under ROC curve = 0.8405S
en
sitiv
ity
1 - Specificity0.00 0.25 0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
![Page 49: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/49.jpg)
Main Point 2ROC Curve Describes the Test,
Not the Patient
• Describes the test’s ability to discriminate between D+ and D- individuals
• Not particularly useful in interpreting a test result for a given patient
![Page 50: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/50.jpg)
ROC Curve Describes the Test, Not the Patient
Clinical Scenario
Synovial WBC count = 48,000
Synovial WBC count = 128,000
![Page 51: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/51.jpg)
Synovial WBC count = 48,000
![Page 52: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/52.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1 - Specificity
Sen
sitiv
ity
Cutoff > ∞
Cutoff > 100k
Cutoff > 50k
Cutoff > 25k
Cutoff ≥ 0
![Page 53: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/53.jpg)
Sensitivity, Specificity, LR(+), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3
Different Cutoffs
WBC (/uL) Sensitivity Specificity LR+ LR-
>100,000 29% 99% 29.0 0.7
>50,000 62% 92% 7.8 0.4
>25,000 77% 73% 2.9 0.3
Synovial WBC Count = 48,000/uL
Which LR should we use?
![Page 54: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/54.jpg)
Likelihood Ratios
LR(+) = Sensitivity/(1 – Specificity) = P(+|D+)/(1-P(-|D-)) = P(+|D+)/P(+|D-)
LR(-) = (1 – Sensitivity)/Specificity = (1-P(+|D+))/P(-|D-) = P(-|D+)/P(-|D-)
![Page 55: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/55.jpg)
Likelihood Ratios
LR(result) = P(result|D+)/P(result|D-)
P(Result) in patient WITH disease
------------------------------------------------------
P(Result) in patients WITHOUT disease
![Page 56: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/56.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
0 - 25,000 >25,000-50,000
>50,000-100,000
>100,000
No Septic Arthritis
Septic Arthritis
Likelihood RatiosThe ratio of the height of the D+ distribution to the height of the D- distribution
15%19%
LR = 15%/19% = 0.8
![Page 57: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/57.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1 - Specificity
Sen
sitiv
ity
> 50k
> 25k
15%
19%Slope = 15%/19% =0.8
![Page 58: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/58.jpg)
Likelihood Ratio
WBC (/uL) Interval % of D+ % of D-
Interval LR
>100,000 29% 1% 29.0
>50,000-100,000 33% 7% 4.7
>25,000-50,000 15% 19% 0.8
0 - 25,000 23% 73% 0.3
![Page 59: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/59.jpg)
Common Mistake
When given an “ROC Table,” it is tempting to calculate an LR(+) or LR(-) as if the test were “dichotomized” at a particular cutoff.
Example: LR(+,25,000) = 77%/27% = 2.9
This is NOT the LR of a particular result (e.g. WBC >25,000 and ≤ 50,000); it is the LR(+) if you divide “+” from “-” at 25,000.
![Page 60: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/60.jpg)
WBC (/uL) Sensitivity Specificity LR+ LR-
>100,000 29% 99% 29.0 0.7
>50,000 62% 92% 7.8 0.4
>25,000 77% 73% 2.9 0.3
Common Mistake
![Page 61: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/61.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1 - Specificity
Sen
sit
ivit
y
Slope = 77%/27% = 2.9
Slope = 0.8
27%
77%
> 25,000
Common Mistake
![Page 62: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/62.jpg)
Main Point 3 Likelihood Ratio
P(Result) in patient WITH disease
------------------------------------------------------
P(Result) in patients WITHOUT disease
Slope of ROC Curve
Do not calculate an LR(+) or LR(-) for a multilevel test.
![Page 63: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/63.jpg)
Clinical ScenarioSynovial WBC = 48,000/uL
Pre-test prob: 0.38
Pre-test odds: 0.38/0.62 = 0.61
LR(WBC btw 25,000 and 50,000) = 0.8
Post-Test Odds = Pre-Test Odds x LR(48)
= 0.61 x 0.8 = 0.49
Post-Test prob = 0.49/(0.49+1) = 0.33
![Page 64: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/64.jpg)
Clinical ScenarioSynovial WBC = 128,000/uL
Pre-test prob: 0.38
Pre-test odds: 0.38/0.62 = 0.61
LR(128,000/uL) = 29
Post-Test Odds = Pre-Test Odds x LR(128)
= 0.61 x 29 = 17.8
Post-Test prob = 17.8/(17.8+1) = 0.95
![Page 65: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/65.jpg)
Clinical Scenario
WBC = 48,000/uL Post-Test Prob = 0.33
WBC = 128,000/uL Post-Test Prob = 0.95
(Recall that dichotomizing the WBC with a fixed cutpoint of 25,000/uL meant that WBC = 48,000/uL would be treated the same as WBC = 128,000/uL and post-test prob = 0.64)
![Page 66: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/66.jpg)
Main Point 4Bayes’s Rule
Pre-Test Odds x LR(result) = Post-Test Odds
What you knew before + What you learned = What you know now
![Page 67: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/67.jpg)
Summary
1) Dichotomizing a multi-level test by choosing a fixed cutpoint reduces the value of the test.
2) The ROC curve summarizes the discriminatory ability of the test.
3) LR(result) = P(result|D+)/P(result|D-) = Slope of ROC Curve
(NOTE: Do not calculate an LR(+) or LR(-) for a multilevel test.)
4) Pre-Test Odds x LR(result) = Post-Test Odds
![Page 68: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/68.jpg)
Calculating the c StatisticIn the “walking man” approach to tracing out the ROC curve, the actual values of the test are not important for the shape of the ROC curve or the area under it--only the ranking of the values.
The c statistic for the area under an ROC curve is calculated using the same information as the Wilcoxon Rank Sum statistic (or Mann-Whitney U, which is equivalent) and gives identical P values.
Non-parametric equivalent of the t test statistic comparing two means.
![Page 69: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/69.jpg)
Septic Arthritis No Septic Arthritis
128
92
71
64
48
37 37
24
23
12
12
8
7
6
0
![Page 70: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/70.jpg)
0
1
2
3
4
5
0 1 2 3 4 5 6 7 8 9 10
D-
D+
Boxes under Curve = 43.5
Total Boxes = 50
Area Under Curve = 43.5/50 = 0.87
![Page 71: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/71.jpg)
BACTEREMIA NO BACTEREMIA
1
2
3
4
5
6.5 6.5
8
9
10.5
10.5
12
13
14
15
S = 21.5
Replace Test Results with Ranks
![Page 72: Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.](https://reader035.fdocuments.us/reader035/viewer/2022062722/56649f315503460f94c4cca6/html5/thumbnails/72.jpg)
S = 21.5Smin = d(d+1)/2 = 5(6)/2 = 15Smax = dn + Smin = 5(10) + 15 = 65
C = (Smax – S) / (Smax – Smin)* = (65 – 21.5) / (65 – 15) = 43.5/50 = 0.87* Smax – Smin = dn
Calculating the C Statistic