CURS 12. Analiza Curbelor ROC

7/28/2019 CURS 12. Analiza Curbelor ROC

1/23

UNIVERSITATEA BABE-BOLYAICLUJ-NAPOCA

FACULTATEA DE PSIHOLOGIE I TIINE ALE EDUCAIEICatedra de Psihologie Clinic i Psihoterapie

ANALIZA CURBELOR ROC

CS III dr. Sebastian Pintea


2/23

Introducere

ROC (Receiver Operating Characteristic) Analysis

Utilizat n analiza proprietilor diagnostice ale testelor(modul n care testele discrimineaz ntre populaia clinici non-clinic)

Rezultatele la test se raporteaz la rezultatul unui diagnosticvalid (golden standard)


3/23

IntroducereIstoricul utilizrilor

n domeniul militar, n analiza imaginilor radar (WW II) n domeniul medical, ncepnd din anii 60 (mai ales n imagistica medical) n chimie (ex. Dac un marker proteic este prezent sau nu)

n psihologia clinic: n evaluarea unor teste diagnostice sau de screening pt:

Dificulti n comprehensiunea limbajului (Shapiro, Solari & Petscher, 2008) Dizabiliti neuropsihologice (Horwitz et al., 2008; O'Brien et al., 2007) Depresie (Benazzi, 2008; Serrano-Duenas & Serrano, 2008; Stafford, Berk &

Jackson, 2007; Ballesteros et al., 2007; Walsh et al., 2006) Tulburare obsesiv-compulsiv (Ivarsson & Larsson, 2008)

Tulburare bipolar (Parker et al., 2008) Suicid(Jokinen, Nordstrom & Nordstrom, 2008) Demen (Chiu et al., 2008; Giaquinto & Parnetti, 2006) Risul de dropoutn diferite intervenii precum terapia cognitiv-

comportamental pentru insomnie (Ong, Kuo & Manber, 2008)


4/23

Indicatori ai curbelor ROCFig. 1.Patru posibile categorii de subieci atunci cnd intersectm un diagnostic valid

cu un clasificator


5/23

Tabelul 1.The confusion matrix

DIAGNOSTIC

TEST Pozitiv Negativ Total

Pozitiv TP FP T+

Negativ FN TN T-

Total D+ D- n

(1) Sensitivity = TP/D+(2) Specificity = TN/D-(3)Positive likelihood ratio = Sensitivity / (1-Specificity)(4)Negative likelihood ratio = (1-Sensitivity) / Specificity(5)Positive predictive value = TP/T+(6)Negative predictive value = TN/T-

(7)Accuracy = (TP+TN)/n


6/23

Spaiul ROC (interpretarea punctelor)

1.00.80.60.40.20.0

1 - Specificity

1.0

0.8

0.6

0.4

0.2

0.0

Sensitivity


7/23

Utilitatea analizei curbelor ROC

Determinarea capacitii unui test de adscrimina ntre grupuri (+, -)

Alegerea unei valori prag (cut-off point) optim Compararea performanei a dou sau mai multeteste


8/23

Determinarea capacitii unui test de adiscrimina ntre grupuri

AUCarea under the curve Cu ct AUC corespunztoare unui test este mai mare, cu att

performana lui discriminativ e mai bun Diagonala spaiului ROC graficul unui test care determin cele dou

categorii diagnostice n mod aleator H0: AUC= 0.50 Streiner and Cairney (2007):

AUC ntre 0.50 i 0.70: acuratee redus a testului AUC ntre 0.70 i 0.90: acuratee moderat AUC peste 0.90: acuratee ridicat


9/23


10/23

Alegerea unei valori prag (cut-off point) optim

Valoarea prag optim este dat de cel mai nord-vestic puncttrasat de curba testului n spaiul ROC

Valoarea prag optim este cea care maximizeaz suma TP +

TN n decizie, se ia n calcul i costul unui fals pozitiv, i costul

unui fals negativ (exemple?) Alte studii recomand:

Specificitate minim de 95% (Westin, 2001) n screening, senzitivitate minim de 80% (Sharifi et al., 2008)

(puterea testului)


11/23


12/23

Compararea performanei a dou saumai multe teste

Compararea ca performanglobal

Compararea pe anumiteintervale de senzitivitate sauspecificitate

Compararea la valorile lor prag

A

B


13/23

Compararea caperforman

global


14/23

Softuri pentru analiza curbelor ROC

AccuROC, Analyse-it, CMDT, GraphROC, MedCalc, mROC, ROCKITand SPSS

Stephan et al. (2003): analiza comparativ a softurilor

Concluziile autorilor:

Doar Analyse-It, AccuROC i MedCalc au dovedit o performanbun

Fiecare soft are propriile neajunsuri

Doar GraphROCpoate s compare curbe la o anumit valoare asenzitivitii sau specificitii

O analiz ROC adecvat inclusiv cu reprezentare grafic nu poate fifcut cu un singur soft

Autorii recomand Analyse-It, AccuROC i MedCalc, dar i acesteacu anumite limitri


15/23

ROC ANALYSIS: AN APPLICATION IN

CLINICAL PSYCHOLOGY

Depression (DSM)

BDI vs ATQObjectives

(1) to evaluate the diagnostic performance for BDI andATQ

(2) to establish the optimal cut-off point of each measure

(3) to test if there is a difference between the two scalesregarding their mean diagnostic performance


16/23

Sample

Data were pooled from participants in two separate studies. Thesample consisted of 50 patients aged between 12 and 18 who metcriteria for MDD according to the DSM-IV. Inclusion criteria requiredthat patients (children and adolescents aged from 12 to 18) meetcriteria for current principal diagnosis of MDD as per the Diagnosticand Statistical Manual of Mental Disorders (4th edition, text revision;

DSM-IV-TR, American Psychiatric Association, 2000). Exclusioncriteria included a number of concurent psychiatric disorders, currentsubstance abuse, mental retardation, organic brain sindrome; we alsoexcluded participants who were in some concurent form of

psychotherapy, who were receiving psychothropic medication or who

needed to be hospitalized because of imminent suicidal risk. Patientswere recruited by local advertisements and by referrals from clinicswithin the Pediatric Psychiatry Clinic in Cluj-Napoca from March2007 until June 2008. A number of 50 other voluntary adolescentsfrom several high schools in Cluj-Napoca were also included.


17/23

Procedure

Patients were assessed prior to random assignmentto three treatment groups [(1) cognitive-behavioralpsychotherapy, (2) medication; (3) cognitive-

behavioral psychotherapy and medication]. Afterdetailed description of the study to all participants,informed consent was obtained. Screeninginstruments were completed individually before

entering treatment and afterwards for 16 weeks untilall treatment conditions are completed.


18/23

Measures

Beck Depression Inventory (BDI; Beck et al., 1979) is a 21-item self-report inventory measuring current characteristicsymptoms of depression (e.g., sadness, fatigue, socialwithdrawal, irritability, hopelessness etc.).

Automatic Thoughts Questionnaire (ATQ; Hollon &Kendal, 1980) is a 15-item questionnaire assessing the

frequency of negative thoughts experienced by depressives.All items consist of different self related automatic thoughts(e.g. I am worthless; Future is dark; I feel helpless),frequently identified in patients with MDD.


19/23

ResultsTable 2. The AUC for ATQ scale

Area under the ROC curve (AUC) 0.909Standard error 0.0307

95% Confidence interval 0.835 to 0.957

z statistic 13.338

Significance level P (Area=0.5) 0.0001

Area under the ROC curve (AUC) 0.996

Standard error 0.00655


z statistic 75.682

Significance level P (Area=0.5) 0.0001

Table 3. The AUC for BDI scale


20/23

Results

Criterion Sensitivity 95% CI Specificity 95% CI +LR -LR>=15 100.00 92.8 - 100.0 0.00 0.0 - 7.2 1.00

>15 100.00 92.8 - 100.0 10.00 3.4 - 21.8 1.11 0.00

>16 100.00 92.8 - 100.0 14.00 5.8 - 26.7 1.16 0.00

>18 100.00 92.8 - 100.0 16.00 7.2 - 29.1 1.19 0.00

>20 100.00 92.8 - 100.0 22.00 11.5 - 36.0 1.28 0.00

>21 100.00 92.8 - 100.0 24.00 13.1 - 38.2 1.32 0.00

>23 100.00 92.8 - 100.0 32.00 19.5 - 46.7 1.47 0.00

>24 100.00 92.8 - 100.0 42.00 28.2 - 56.8 1.72 0.00

>25 100.00 92.8 - 100.0 48.00 33.7 - 62.6 1.92 0.00

>26 100.00 92.8 - 100.0 50.00 35.5 - 64.5 2.00 0.00

>27 98.00 89.3 - 99.7 58.00 43.2 - 71.8 2.33 0.034

>28 98.00 89.3 - 99.7 60.00 45.2 - 73.6 2.45 0.033

>29 96.00 86.3 - 99.4 62.00 47.2 - 75.3 2.53 0.065

>31 96.00 86.3 - 99.4 66.00 51.2 - 78.8 2.82 0.061

>33 94.00 83.4 - 98.7 68.00 53.3 - 80.5 2.94 0.088

>34* 94.00 83.4 - 98.7 70.00 55.4 - 82.1 3.13 0.086

>35 90.00 78.2 - 96.6 70.00 55.4 - 82.1 3.00 0.14

Table 4. Criterion values and coordinates of the ROC curve for ATQ


21/23

Results

Criterion Sensitivity 95% CI Specificity 95% CI +LR -LR

>=0 100.00 92.8 - 100.0 0.00 0.0 - 7.2 1.00

>0 100.00 92.8 - 100.0 12.00 4.6 - 24.3 1.14 0.00

>1 100.00 92.8 - 100.0 20.00 10.0 - 33.7 1.25 0.00

>2 100.00 92.8 - 100.0 28.00 16.2 - 42.5 1.39 0.00

>3 100.00 92.8 - 100.0 40.00 26.4 - 54.8 1.67 0.00

>4 100.00 92.8 - 100.0 42.00 28.2 - 56.8 1.72 0.00

>5 100.00 92.8 - 100.0 56.00 41.3 - 70.0 2.27 0.00>6 100.00 92.8 - 100.0 60.00 45.2 - 73.6 2.50 0.00

>8 100.00 92.8 - 100.0 64.00 49.2 - 77.1 2.78 0.00

>9 100.00 92.8 - 100.0 70.00 55.4 - 82.1 3.33 0.00

>10 100.00 92.8 - 100.0 82.00 68.6 - 91.4 5.56 0.00

>14 100.00 92.8 - 100.0 84.00 70.9 - 92.8 6.25 0.00

>15 100.00 92.8 - 100.0 88.00 75.7 - 95.4 8.33 0.00>17 100.00 92.8 - 100.0 90.00 78.2 - 96.6 10.00 0.00

>18 100.00 92.8 - 100.0 94.00 83.4 - 98.7 16.67 0.00

>21* 100.00 92.8 - 100.0 96.00 86.3 - 99.4 25.00 0.00

>22 92.00 80.7 - 97.7 96.00 86.3 - 99.4 23.00 0.083

>23 90.00 78.2 - 96.6 98.00 89.3 - 99.7 45.00 0.10

>24 86.00 73.3 - 94.2 100.00 92.8 - 100.0 0.14

Table 5. Criterion values and coordinates of the ROC curve for BDI scale


22/23

Results

Table 6. Pairwise comparison of ATQ and BDI AUC's

ATQ ~ BDI

Difference between areas 0.0868

Standard error 0.0299


z statistic 2.907

Significance level P = 0.004


23/23

Results

1.00.80.60.40.20.0

1 - Specificity

1.0

0.8

0.6

0.4

0.2

0.0

Se

nsi

tivi

ty

Reference Line

BDI

ATQ

Source of the Curve

Figure 4. The ROC curves of ATQ and BDI

CURS 12. Analiza Curbelor ROC

Documents

Transcript of CURS 12. Analiza Curbelor ROC