Post on 03-Apr-2018
7/28/2019 CURS 12. Analiza Curbelor ROC
1/23
UNIVERSITATEA BABE-BOLYAICLUJ-NAPOCA
FACULTATEA DE PSIHOLOGIE I TIINE ALE EDUCAIEICatedra de Psihologie Clinic i Psihoterapie
ANALIZA CURBELOR ROC
CS III dr. Sebastian Pintea
7/28/2019 CURS 12. Analiza Curbelor ROC
2/23
Introducere
ROC (Receiver Operating Characteristic) Analysis
Utilizat n analiza proprietilor diagnostice ale testelor(modul n care testele discrimineaz ntre populaia clinici non-clinic)
Rezultatele la test se raporteaz la rezultatul unui diagnosticvalid (golden standard)
7/28/2019 CURS 12. Analiza Curbelor ROC
3/23
IntroducereIstoricul utilizrilor
n domeniul militar, n analiza imaginilor radar (WW II) n domeniul medical, ncepnd din anii 60 (mai ales n imagistica medical) n chimie (ex. Dac un marker proteic este prezent sau nu)
n psihologia clinic: n evaluarea unor teste diagnostice sau de screening pt:
Dificulti n comprehensiunea limbajului (Shapiro, Solari & Petscher, 2008) Dizabiliti neuropsihologice (Horwitz et al., 2008; O'Brien et al., 2007) Depresie (Benazzi, 2008; Serrano-Duenas & Serrano, 2008; Stafford, Berk &
Jackson, 2007; Ballesteros et al., 2007; Walsh et al., 2006) Tulburare obsesiv-compulsiv (Ivarsson & Larsson, 2008)
Tulburare bipolar (Parker et al., 2008) Suicid(Jokinen, Nordstrom & Nordstrom, 2008) Demen (Chiu et al., 2008; Giaquinto & Parnetti, 2006) Risul de dropoutn diferite intervenii precum terapia cognitiv-
comportamental pentru insomnie (Ong, Kuo & Manber, 2008)
7/28/2019 CURS 12. Analiza Curbelor ROC
4/23
Indicatori ai curbelor ROCFig. 1.Patru posibile categorii de subieci atunci cnd intersectm un diagnostic valid
cu un clasificator
7/28/2019 CURS 12. Analiza Curbelor ROC
5/23
Tabelul 1.The confusion matrix
DIAGNOSTIC
TEST Pozitiv Negativ Total
Pozitiv TP FP T+
Negativ FN TN T-
Total D+ D- n
(1) Sensitivity = TP/D+(2) Specificity = TN/D-(3)Positive likelihood ratio = Sensitivity / (1-Specificity)(4)Negative likelihood ratio = (1-Sensitivity) / Specificity(5)Positive predictive value = TP/T+(6)Negative predictive value = TN/T-
(7)Accuracy = (TP+TN)/n
7/28/2019 CURS 12. Analiza Curbelor ROC
6/23
Spaiul ROC (interpretarea punctelor)
1.00.80.60.40.20.0
1 - Specificity
1.0
0.8
0.6
0.4
0.2
0.0
Sensitivity
7/28/2019 CURS 12. Analiza Curbelor ROC
7/23
Utilitatea analizei curbelor ROC
Determinarea capacitii unui test de adscrimina ntre grupuri (+, -)
Alegerea unei valori prag (cut-off point) optim Compararea performanei a dou sau mai multeteste
7/28/2019 CURS 12. Analiza Curbelor ROC
8/23
Determinarea capacitii unui test de adiscrimina ntre grupuri
AUCarea under the curve Cu ct AUC corespunztoare unui test este mai mare, cu att
performana lui discriminativ e mai bun Diagonala spaiului ROC graficul unui test care determin cele dou
categorii diagnostice n mod aleator H0: AUC= 0.50 Streiner and Cairney (2007):
AUC ntre 0.50 i 0.70: acuratee redus a testului AUC ntre 0.70 i 0.90: acuratee moderat AUC peste 0.90: acuratee ridicat
7/28/2019 CURS 12. Analiza Curbelor ROC
9/23
7/28/2019 CURS 12. Analiza Curbelor ROC
10/23
Alegerea unei valori prag (cut-off point) optim
Valoarea prag optim este dat de cel mai nord-vestic puncttrasat de curba testului n spaiul ROC
Valoarea prag optim este cea care maximizeaz suma TP +
TN n decizie, se ia n calcul i costul unui fals pozitiv, i costul
unui fals negativ (exemple?) Alte studii recomand:
Specificitate minim de 95% (Westin, 2001) n screening, senzitivitate minim de 80% (Sharifi et al., 2008)
(puterea testului)
7/28/2019 CURS 12. Analiza Curbelor ROC
11/23
7/28/2019 CURS 12. Analiza Curbelor ROC
12/23
Compararea performanei a dou saumai multe teste
Compararea ca performanglobal
Compararea pe anumiteintervale de senzitivitate sauspecificitate
Compararea la valorile lor prag
A
B
7/28/2019 CURS 12. Analiza Curbelor ROC
13/23
Compararea caperforman
global
7/28/2019 CURS 12. Analiza Curbelor ROC
14/23
Softuri pentru analiza curbelor ROC
AccuROC, Analyse-it, CMDT, GraphROC, MedCalc, mROC, ROCKITand SPSS
Stephan et al. (2003): analiza comparativ a softurilor
Concluziile autorilor:
Doar Analyse-It, AccuROC i MedCalc au dovedit o performanbun
Fiecare soft are propriile neajunsuri
Doar GraphROCpoate s compare curbe la o anumit valoare asenzitivitii sau specificitii
O analiz ROC adecvat inclusiv cu reprezentare grafic nu poate fifcut cu un singur soft
Autorii recomand Analyse-It, AccuROC i MedCalc, dar i acesteacu anumite limitri
7/28/2019 CURS 12. Analiza Curbelor ROC
15/23
ROC ANALYSIS: AN APPLICATION IN
CLINICAL PSYCHOLOGY
Depression (DSM)
BDI vs ATQObjectives
(1) to evaluate the diagnostic performance for BDI andATQ
(2) to establish the optimal cut-off point of each measure
(3) to test if there is a difference between the two scalesregarding their mean diagnostic performance
7/28/2019 CURS 12. Analiza Curbelor ROC
16/23
Sample
Data were pooled from participants in two separate studies. Thesample consisted of 50 patients aged between 12 and 18 who metcriteria for MDD according to the DSM-IV. Inclusion criteria requiredthat patients (children and adolescents aged from 12 to 18) meetcriteria for current principal diagnosis of MDD as per the Diagnosticand Statistical Manual of Mental Disorders (4th edition, text revision;
DSM-IV-TR, American Psychiatric Association, 2000). Exclusioncriteria included a number of concurent psychiatric disorders, currentsubstance abuse, mental retardation, organic brain sindrome; we alsoexcluded participants who were in some concurent form of
psychotherapy, who were receiving psychothropic medication or who
needed to be hospitalized because of imminent suicidal risk. Patientswere recruited by local advertisements and by referrals from clinicswithin the Pediatric Psychiatry Clinic in Cluj-Napoca from March2007 until June 2008. A number of 50 other voluntary adolescentsfrom several high schools in Cluj-Napoca were also included.
7/28/2019 CURS 12. Analiza Curbelor ROC
17/23
Procedure
Patients were assessed prior to random assignmentto three treatment groups [(1) cognitive-behavioralpsychotherapy, (2) medication; (3) cognitive-
behavioral psychotherapy and medication]. Afterdetailed description of the study to all participants,informed consent was obtained. Screeninginstruments were completed individually before
entering treatment and afterwards for 16 weeks untilall treatment conditions are completed.
7/28/2019 CURS 12. Analiza Curbelor ROC
18/23
Measures
Beck Depression Inventory (BDI; Beck et al., 1979) is a 21-item self-report inventory measuring current characteristicsymptoms of depression (e.g., sadness, fatigue, socialwithdrawal, irritability, hopelessness etc.).
Automatic Thoughts Questionnaire (ATQ; Hollon &Kendal, 1980) is a 15-item questionnaire assessing the
frequency of negative thoughts experienced by depressives.All items consist of different self related automatic thoughts(e.g. I am worthless; Future is dark; I feel helpless),frequently identified in patients with MDD.
7/28/2019 CURS 12. Analiza Curbelor ROC
19/23
ResultsTable 2. The AUC for ATQ scale
Area under the ROC curve (AUC) 0.909Standard error 0.0307
95% Confidence interval 0.835 to 0.957
z statistic 13.338
Significance level P (Area=0.5) 0.0001
Area under the ROC curve (AUC) 0.996
Standard error 0.00655
95% Confidence interval 0.955 to 0.996
z statistic 75.682
Significance level P (Area=0.5) 0.0001
Table 3. The AUC for BDI scale
7/28/2019 CURS 12. Analiza Curbelor ROC
20/23
Results
Criterion Sensitivity 95% CI Specificity 95% CI +LR -LR>=15 100.00 92.8 - 100.0 0.00 0.0 - 7.2 1.00
>15 100.00 92.8 - 100.0 10.00 3.4 - 21.8 1.11 0.00
>16 100.00 92.8 - 100.0 14.00 5.8 - 26.7 1.16 0.00
>18 100.00 92.8 - 100.0 16.00 7.2 - 29.1 1.19 0.00
>20 100.00 92.8 - 100.0 22.00 11.5 - 36.0 1.28 0.00
>21 100.00 92.8 - 100.0 24.00 13.1 - 38.2 1.32 0.00
>23 100.00 92.8 - 100.0 32.00 19.5 - 46.7 1.47 0.00
>24 100.00 92.8 - 100.0 42.00 28.2 - 56.8 1.72 0.00
>25 100.00 92.8 - 100.0 48.00 33.7 - 62.6 1.92 0.00
>26 100.00 92.8 - 100.0 50.00 35.5 - 64.5 2.00 0.00
>27 98.00 89.3 - 99.7 58.00 43.2 - 71.8 2.33 0.034
>28 98.00 89.3 - 99.7 60.00 45.2 - 73.6 2.45 0.033
>29 96.00 86.3 - 99.4 62.00 47.2 - 75.3 2.53 0.065
>31 96.00 86.3 - 99.4 66.00 51.2 - 78.8 2.82 0.061
>33 94.00 83.4 - 98.7 68.00 53.3 - 80.5 2.94 0.088
>34* 94.00 83.4 - 98.7 70.00 55.4 - 82.1 3.13 0.086
>35 90.00 78.2 - 96.6 70.00 55.4 - 82.1 3.00 0.14
Table 4. Criterion values and coordinates of the ROC curve for ATQ
7/28/2019 CURS 12. Analiza Curbelor ROC
21/23
Results
Criterion Sensitivity 95% CI Specificity 95% CI +LR -LR
>=0 100.00 92.8 - 100.0 0.00 0.0 - 7.2 1.00
>0 100.00 92.8 - 100.0 12.00 4.6 - 24.3 1.14 0.00
>1 100.00 92.8 - 100.0 20.00 10.0 - 33.7 1.25 0.00
>2 100.00 92.8 - 100.0 28.00 16.2 - 42.5 1.39 0.00
>3 100.00 92.8 - 100.0 40.00 26.4 - 54.8 1.67 0.00
>4 100.00 92.8 - 100.0 42.00 28.2 - 56.8 1.72 0.00
>5 100.00 92.8 - 100.0 56.00 41.3 - 70.0 2.27 0.00>6 100.00 92.8 - 100.0 60.00 45.2 - 73.6 2.50 0.00
>8 100.00 92.8 - 100.0 64.00 49.2 - 77.1 2.78 0.00
>9 100.00 92.8 - 100.0 70.00 55.4 - 82.1 3.33 0.00
>10 100.00 92.8 - 100.0 82.00 68.6 - 91.4 5.56 0.00
>14 100.00 92.8 - 100.0 84.00 70.9 - 92.8 6.25 0.00
>15 100.00 92.8 - 100.0 88.00 75.7 - 95.4 8.33 0.00>17 100.00 92.8 - 100.0 90.00 78.2 - 96.6 10.00 0.00
>18 100.00 92.8 - 100.0 94.00 83.4 - 98.7 16.67 0.00
>21* 100.00 92.8 - 100.0 96.00 86.3 - 99.4 25.00 0.00
>22 92.00 80.7 - 97.7 96.00 86.3 - 99.4 23.00 0.083
>23 90.00 78.2 - 96.6 98.00 89.3 - 99.7 45.00 0.10
>24 86.00 73.3 - 94.2 100.00 92.8 - 100.0 0.14
Table 5. Criterion values and coordinates of the ROC curve for BDI scale
7/28/2019 CURS 12. Analiza Curbelor ROC
22/23
Results
Table 6. Pairwise comparison of ATQ and BDI AUC's
ATQ ~ BDI
Difference between areas 0.0868
Standard error 0.0299
95% Confidence interval 0.0283 to 0.145
z statistic 2.907
Significance level P = 0.004
7/28/2019 CURS 12. Analiza Curbelor ROC
23/23
Results
1.00.80.60.40.20.0
1 - Specificity
1.0
0.8
0.6
0.4
0.2
0.0
Se
nsi
tivi
ty
Reference Line
BDI
ATQ
Source of the Curve
Figure 4. The ROC curves of ATQ and BDI