InterobserverVariabilityintheInterpretationofMyocardialImageswithIc...

9
CLINICAL SCIENCES The present study was designed to determine the possible bias of experience on the correct interpretation of Tc-99m phosphate myocardial imaging in patients with acute precordial chest pain from diverse causes. MATERIALS AND METHODS Included are the results obtained in a consecutive Se ries of 250 myocardial scans performed with Tc-99m- labeled phosphates, in 226 patients ( I 67 male, 59 fe male) with acute precordial chest pain. The average patient's age was 54.2 yr (range, 17—87). All studies were performed during the first 5 days after the onset of acute chest pain. One hundred twenty seven studies (50.8%) were done with Tc-99rn methylene diphosphonate (MDP), and 123 (49,2%) with Tc-99m(Sn)pyrophos Scintigraphic visualization of an acute myocardial infarction (AMI) with technetium-99m-labeled phos phates has become a common diagnostic procedure in nuclear medicine and a valuable aid in a workup for suspected myocardial infarction (1,2). However, this procedure is frequently hampered by the uptake of the tracer by cardiac processes differing from AMI (3—i 1), which are a potential source of an erroneous diagnosis. Furthermore, visual detection of regional density dif ferences in a scintigram depends greatly on the experi ence ofthe observer (12,13). Received Nov. 20, 1978; revisionacceptedJuly 6, 1979. Forreprintscontact: AlfredoCuarón, Departamento deMedicina N uclear, Instituto Nacional de Cardiolog@aDr. lgnacio Chavez, Cu bkulo B-408, Juan Badiano I, Mexico 22, D.F. Mexico. Volume 21, Number 1 DIAGNOSTIC NUCLEAR MEDICINE InterobserverVariability in the Interpretationof Myocardial Images with Ic 99m-Labeled Diphosphonateand Pyrophosphate Alfredo Cuarôn,Alberto P.Acero, ManuelCárdenas, David Huerta, Alfredo Rodriguez,and Rolando de Garay InstitutoNaciona! deCardiologia Dr.!gnacioChavez, MexicoCity,Mexico Without prior knowledge of the significant clinical data, sIx observers have Inde pendently evaluated a consecutive series of 250 myocardial scans mane with Tc 99m-Iabeled phosphates: 127 with MDP and 123 wIth PPI. Of the 226 patIents, all having acute precordial chest pain, 169 were shown to have acute myocardlal in farctlon while 57 suffered acute distress from other causes. The six observers, varying in their experience with nuclear medicine, compared the Intensity of up take in the heart with that In bone, and rated their ImpressIon of a â€oepositive― Image by a six-category scale—that is, one with five criterion levels. Results were cx pressed as receiver operating characteristic (ROC) curves, from which the optimal individual criterion level for each observer was determined. We found very high in terobserver variability in the perception of the shades of myocardial poncentration, although they were based on strict and apparently objective criteria. This variabiIi ty has a direct influence on the overall performance of each observer. In every in stance, PPI was demonstrated to be a better tracer than MDP for myocardial imag ing. The bias of the experience, visual perception, and psychology of the observer at the time of the reading of the images seems to be significant, as is the presence of uncorrected visual defects. These results justify the settIng of specIal programs to evaluate periodically the perfor@nanceofevery physician who interprets studies, to establish his optimal individpal criterion level instead of using a fixed criterion level to decide whether an image is â€oepositive.― J NucIMed21: 1—9, 1980

Transcript of InterobserverVariabilityintheInterpretationofMyocardialImageswithIc...

Page 1: InterobserverVariabilityintheInterpretationofMyocardialImageswithIc …jnm.snmjournals.org/content/21/1/1.full.pdf · 2006. 12. 19. · TABLEFOREACH1-A.ANALYSISOFRESULTS CONSECUTIVEMYOCARDIALSCANSOBSERVERIN127

CLINICAL SCIENCES

The present study was designed to determine thepossible bias of experience on the correct interpretationof Tc-99m phosphate myocardial imaging in patientswith acute precordial chest pain from diverse causes.

MATERIALS AND METHODS

Included are the results obtained in a consecutive Series of 250 myocardial scans performed with Tc-99m-labeled phosphates, in 226 patients ( I67 male, 59 female) with acute precordial chest pain. The averagepatient's age was 54.2 yr (range, 17—87).All studies wereperformed during the first 5 days after the onset of acutechest pain. One hundred twenty seven studies (50.8%)were done with Tc-99rn methylene diphosphonate(MDP), and 123 (49,2%) with Tc-99m(Sn)pyrophos

Scintigraphic visualization of an acute myocardialinfarction (AMI) with technetium-99m-labeled phosphates has become a common diagnostic procedure innuclear medicine and a valuable aid in a workup forsuspected myocardial infarction (1,2). However, thisprocedure is frequently hampered by the uptake of thetracer by cardiac processes differing from AMI (3—i1),which are a potential source of an erroneous diagnosis.Furthermore, visual detection of regional density differences in a scintigram depends greatly on the experience ofthe observer (12,13).

Received Nov. 20, 1978; revisionacceptedJuly 6, 1979.Forreprintscontact:AlfredoCuarón,DepartamentodeMedicina

N uclear, Instituto Nacional de Cardiolog@aDr. lgnacio Chavez, Cubkulo B-408, Juan Badiano I, Mexico 22, D.F. Mexico.

Volume 21, Number 1

DIAGNOSTIC NUCLEAR MEDICINE

InterobserverVariability in the Interpretationof Myocardial Imageswith Ic

99m-Labeled Diphosphonateand Pyrophosphate

Alfredo Cuarôn,Alberto P. Acero, ManuelCárdenas,David Huerta, Alfredo Rodriguez,and Rolandode Garay

InstitutoNaciona!deCardiologiaDr.!gnacioChavez,MexicoCity,Mexico

Without prior knowledge of the significant clinical data, sIx observers have Independently evaluated a consecutive series of 250 myocardial scans mane with Tc99m-Iabeled phosphates: 127 with MDP and 123 wIth PPI. Of the 226 patIents, allhaving acute precordial chest pain, 169 were shown to have acute myocardlal infarctlon while 57 suffered acute distress from other causes. The six observers,varying in their experience with nuclear medicine, compared the Intensity of uptake in the heart with that In bone, and rated their ImpressIon of a “positive―Imageby a six-category scale—that is, one with five criterion levels. Results were cxpressed as receiver operating characteristic (ROC) curves, from which the optimalindividual criterion level for each observer was determined. We found very high interobserver variability in the perception of the shades of myocardial poncentration,although they were based on strict and apparently objective criteria. This variabiIity has a direct influence on the overall performance of each observer. In every instance, PPI was demonstrated to be a better tracer than MDP for myocardial imaging. The bias of the experience, visual perception, and psychology of the observerat the time of the reading of the images seems to be significant, as is the presenceof uncorrected visual defects. These results justify the settIng of specIal programsto evaluate periodically the perfor@nanceof every physician who interprets studies,to establish his optimal individpal criterion level instead of using a fixed criterionlevel to decide whether an image is “positive.―

J NucIMed21: 1—9,1980

Page 2: InterobserverVariabilityintheInterpretationofMyocardialImageswithIc …jnm.snmjournals.org/content/21/1/1.full.pdf · 2006. 12. 19. · TABLEFOREACH1-A.ANALYSISOFRESULTS CONSECUTIVEMYOCARDIALSCANSOBSERVERIN127

CUARON, ACERO, CARDENAS. HUIiRTA, RODRIGUEZ. AND DEGARAY

phate (PPi). Two hours before imaging, 15 mCi of eithertracer were injected intravenously. Patients were imagedin three projections: anterior, 45°left anterior oblique,and left lateral and were in the imaging room for lessthan 30 mm. During imaging, the patient's electrocardiogram (ECG) was monitored continuously. The scans,each accumulating 500,000 total counts, were displayedwith a scintillation camera and Polaroid film using ahigh-resolution, parallel-hole, I40-keV collimator.

A firm diagnosis was achieved in each patient bymeans of the classical criteria: clinical history, ECG, andserum CPK concentration. The diagnosis of AMI wasbased on at least two of the following criteria.I.Clinical.Sustainedprecordialchestpainunre

sponsive to nitrates, occurring within 24 hr of admission.

2. Serumenzymes.Elevationandsubsequentreturnto normal of at least one, but usually all three, of theseserum enzymes: creatine phosphokinase (CPK), glutamic oxaloacetic transaminase (SGOT), and lacticdehydrogenase (LDH). Blood samples for the enzymedetermination were drawn at admission, then daily forat least 4 consecutive days. Cardiac-specific isoenzymeswere not obtained, and this represents a potential limitation of the diagnostic criteria. However, no patientreceived intramuscular injections or sustained muscletrauma before enzymatic determinations.

3. Electrocardiogram.For transmural infarction, werequired the development of new pathological Qwaves—0.04 sec in duration and at least 25% of theamplitude of the following R wave. For nontransmuralinfarction, T-wave inversion, or evolutionary ST-segmentdepression ( 1.0 mm), or both, without loss of R-wavevoltage or development of new Q waves. ECGs wereobtained at admission and at least daily during the patient's stay in the coronary care unit.

Following these criteria, 169 patients (74.8%) hadsuffered AMI, and 57 (25.2%) had acute distress fromothercauses.

Interpretation of the myocardial images. The intensityof the myocardial concentration of the tracer was ratedfrom 0 to 4+, by using the format proposed by Bermanet al. (14), by which 0 and 1+ are considered withinnormal limits. The 2+ images, with less density in theheart than in the ribs, were divided into two separategroups: 2D for a diffuse pattern, considered by Bermanas an equivocal result, and 2F for a focal pattern, considered as positive for AMI. The 3+ rating indicatesequal densities in heart and ribs; 4+ finds it higher in theheart. Both are considered positive for AMI.

Observers.All imageswere independentlyanalyzedby six observers with different degrees of experience ininterpreting scintigraphic images. None had knowledgeof the patient's clinical history, ECG, CPK serum concentration, or any other pertinent clinical information.Observers I and 2 were attending physicians with I6 and

4 yr of experiencein nuclearmedicine,respectively.Observers 3 and 4 were attending physicians with 20 and4 yrofexperienceincardiology,respectively,butwithonly18 mo oftrainingingamma imaging.Observers5and 6 were fellows in nuclear medicine with only 3 moof experience.

Method ofanalysis. The observers were instructed torecord their interpretation for each image by rating theintensity of the myocardial uptake of the tracer according to the foregoing six grades, which imply fivecriterion levels for calling an image “positive.―The mostdemanding of these levels classifies as “positive―onlythose images with a 4+ myocardial intensity. The mostlax regards all images graded from I+ to 4+ as “positive.

The true-positive fraction (sensitivity), the true-negative fraction (specificity), the false-positive, andfalse-negative fractions, and the accuracy, at each of thefive decision thresholds implied by the six-category scale,were calculated for each observer using a two by twodecision matrix, which relates the presence or absenceofAMI with a binary outcome ofthe imaging procedure:positive or negative (15).

The optimal criterion level for each observer was determined as the one in which a shift to a more lax critenon level causes a larger increment in false-positivefraction than in true-positive fraction for the observer,and a shift to a stricter criterion level produces a greaterdecrement in true-positive fraction than in false-positivefraction.

The resulting true-positive and false-positive fractions,at the five decision thresholds, were plotted for eachobserver in a separate coordinate space for every observergroup: nuclear medicine physicians, cardiologists, orfellows. Receiver operating characteristic (ROC) curveswere fitted by plotting the operating points on doubleprobability paper (15) and drawing a straight linethrough the points, visually dividing the errors. The resuIting lines for each observer were then transferred tothe corresponding coordinate space. Error bars representing the square root of binomial variance were fittedto all operating points in order to obtain a qualitativeimpression of the significance of the curve separation(15). Performanceof each observerwith the two radiotracers employed (MDP and PPi) was separatelyevaluated by ROC curve. analysis.

Analysis of data on the basis of the sign test with twotails was done, according to the values obtained by eachobserver when using his own optimal criterion level.

RESULTS

The analysis of results for each observer is shown inTable I , A and B. It demonstrates by itself the great interobserver variability in distinguishing the differentshades of myocardial concentration of radiophosphates

2 THE JOURNAL OF NUCLEAR MEDICINE

Page 3: InterobserverVariabilityintheInterpretationofMyocardialImageswithIc …jnm.snmjournals.org/content/21/1/1.full.pdf · 2006. 12. 19. · TABLEFOREACH1-A.ANALYSISOFRESULTS CONSECUTIVEMYOCARDIALSCANSOBSERVERIN127

TABLE 1-A.ANALYSIS OF RESULTSFOREACHOBSERVER IN 127CONSECUTIVEMYOCARDIALSCANS

(Tc-99m METHYLENEDIPHOSPHONATE)ObserverPatterns

and intensity of myocardialImage0

1+ 2D 2F 3+ 4+

TABLE 1-B. ANALYSIS OF RESULTSFOREACHOBSERVER IN 123CONSECUTIVEMYOCARDIAL

SCANS (Tc-99mSTANNOUSPYROPHOSPHATE)Pattern

and Intensity of myocardialimageObserver0 1+ 20 2F 3+ 4+

CLINICAL SCIENCESDIAGNOSTICNUCLEARMEDICINE

as 4+, whereas Observer 4 rated only two as grade 0 and22as4+. In patientswithoutAMI, theobserversdisagreed similarly in their grading of images with eithertracer.

Table 2 (A and B) lists the true- and false-positivefractions, the true- and false-negative fractions, and theaccuracy generated by the six observers with both radiophosphates, at five different criterion levels. Theunderlined values designate the optimal individual critenon level estimated for each observer. When usingMDP, Observers I, 3, and 6 agreed that a 2+ myocardial intensity with a focal pattern (2F) indicates apositive result for AMI. On the other hand, Observers2and5showedamorelaxoptimalindividualcriterionlevel, defining as positive for AMI those images with a2+ intensitywithadiffusepattern(2D),whereasObserver 4 used a stricter optimal criterion level, considering only 3+ and 4+ as positive for AMI (Table2-A).

When using PPi, Observers 2, 3, and 6 each shifted hisoptimal criterion levels to a stricter one, while observers4 and 5 retained the sameoptimal criterion levelsthatthey showed with MDP (Table 2-B).

The plotting of each true-positive fraction against thecorresponding false-positive fraction allowed us to drawsmooth ROC curves, with five operating points each, forevery observer (Fig. I). These curves represent a cornplete description of the observer performance over a wideand continuous range of criterion levels. The fartherupward and to the left a curve is located, the better is theobserver performance.

The ROC curves shown in Fig. I-A demonstrate abetter performance for Observer I, compared with thatof Observer 2, in the interpretation of myocardial imageswith MDP and PPi; that is to say, it shows a highersensitivity and a higher specificity for Observer I overthe wide and continuous range of criterion levels. Bothnuclear medicine physicians (Fig. I-A) demonstratedbetter performance than the cardiologists (Fig. I-B), andthese performed better than the two fellows with only 3mo oftraining (Fig. I-C). It is also evident that @lltheobservers gave a better performance with PPi than withMDP. Analysis of the figures for each statistical parameter, using the optimal individual criterion level, onthe basis of the sign test with two tails, has shown thatPPi is a better myocardial scanning agent than MDP (p= 0.05).

Table 3 shows the values for sensitivity, specificity,and accuracy generated by each observer at his ownoptimal criterion level. There is a clear trend for the threestatistical parameters to increase with the experience ofthe observer in interpreting nuclear medicine images.Both nuclear medicine physicians attained the highestvalues for accuracy, followed by the cardiologist with I8mo experience in gamma imaging. The fellows (3 mo oftraining) obtained lowest values for accuracy. In every

Withacutemyocardialinfarction(90images)1 14 1 1 41 29 42 24 8 16 24 17 13 11 5 13 18 32 114 7 6 9 20 30 18

5 37 3 11 19 20 06 24 2 14 18 31 1

Withoutacutemyocardialinfarction(37 images)

1 24 4 4 5 0 02 28 5 3 1 0 0

3 11 2 10 6 7 14 11 6 7 9 3 15 26 2 3 4 2 06 14 1 9 7 5 1

Withacutemyocardialinfarction(95images)1 5 1 4 43 36 62 13 2 11 39 26 43 7 2 6 9 54 174 2 2 2 19 48 22

5 29 6 11 13 32 46 6 2 19 9 56 3

Without acute myocardial Infarction (28 images)

1 17 5 3 3 0 02 18 3 5 2 0 03 10 1 6 5 6 0

4 7 5 5 8 3 05 22 2 2 1 1 06 7 2 8 5 6 0

in the 250 images. Even at the extreme grades of myocardial intensity (0 and 4+), the observers were in totaldisagreement. From 90 images obtained with MDP inpatients with AMI, Observer 5 reported 37 images with0 intensityandnonewith a 4+ intensity,whereasObserver 4 reported seven images as 0 and 18 as 4+ (Table1-A). A similar disagreement was observed in 95 imagesobtained with PPi in patients with AMI. In this instance,Observer 5 reported 29 images as 0 intensity, and four

Volume 21, Number 1 3

Page 4: InterobserverVariabilityintheInterpretationofMyocardialImageswithIc …jnm.snmjournals.org/content/21/1/1.full.pdf · 2006. 12. 19. · TABLEFOREACH1-A.ANALYSISOFRESULTS CONSECUTIVEMYOCARDIALSCANSOBSERVERIN127

1+2D2D2F2F2F3+3+3+3+Observer

4+ 4+4+4+4+

CUARON. ACERO. CARDENAS, HUERTA, RODRIGUEZ, AND DEGARAY

True-positive

fraction(sensitivity)1

234

560.044

0.0110.1220.2000.0000.0110.367

0.2000.478

@&@_@‘

0.2170.3560.822

0.46706780.7560.4330.5560.833

06440.8220.8560.5560.7110.844

0.7330.8780.9220.589

0.733True-negative

fraction

(specificity)1

2

34

561.000

1.000

0.9730.9731.0000.9731.000

1.0000.784

0.8920.9460.8380.865

0.9730.6220.6490.83806490.757

08920.3510.4590.7570.4050.649

0.7570.2970.2970.703

0.379False-positive

fractiona1

2

34560.000

0.0000.0270.0270.0000.0270.000

0.000

0.216

0.1080.0540.1620.135

0.027

03780.3510.16203510.243

0.1080.6490.5410.2430.5950.351

0.2430.7030.7030.297

0.622False-negative

fraction1 234560.956

0.9890.8780.8001.0000.9890.633

0.8000.522

Q:@!t0.7830.6440.178

0.5330.3220.2440.56704440.167

0.3560.1780.14404440.2890.156

0.2670.1220.0780.411

0.267Accuracy1

234560.323

0.2990.3700.4250.2910.2910.551

0.4330.5670.6380.4330.4960.835

0.61406610.7240.5510.5830.811

0.7170.6850.7400.6140.6220.787

0.7400.7090.7400.6220.630

. Underlined figures represent the individual observer's optimal criterion level.

instance, diagnostic performance was better when usingPPi, than with MDP.

DISCUSSION

Our results seems to demonstrate a bias on the correctinterpretation of Tc-99m-labeled phosphate myocardialimages, related to the training and visual perception ofthe observer. Although we followed an apparently objective method for the rating of myocardial tracer uptake, comparing it with that in bone (14), we found avery high interobserver variability in the perception ofthe different shades of myocardial concentration (Table

I). This variability exerts a direct influence on the overallperformance of each observer, as is shown by their respective ROC curves (Fig. 2) and the optimal individualcriterion levels (Tables 2 and 3). The best performer witheither MDP or PPi, was Observer 1, with 16 yr experience in nuclear medicine. He was followed by Observer2, with 4 yr formal training in nuclear medicine. It hasturned out since that Observer 2 was wearing faultyglasses, which failed to correct a mild visual defect. Hisperformance might well have been much better had hisglasses provided adequate correction. We are thuswarned of the probably important role of uncorrectedvisual defects in the interpretation ofgamma images.

4 THE JOURNAL OF NUCLEAR MEDICINE

TABLE 2-A. TRUE- AND FALSE-POSITIVEFRACTiONS, TRUE- AND FALSE-NEGATIVE FRACTIONS,AND ACCURACY, AT FIVE CRITERION LEVELS, FOR SIX DIFFERENT OBSERVERS INTERPRETING

127 MYOCARDIAL IMAGES OBTAINED WITH Tc-99m METHYLENE DIPHOSPHONATE

Page 5: InterobserverVariabilityintheInterpretationofMyocardialImageswithIc …jnm.snmjournals.org/content/21/1/1.full.pdf · 2006. 12. 19. · TABLEFOREACH1-A.ANALYSISOFRESULTS CONSECUTIVEMYOCARDIALSCANSOBSERVERIN127

Tc-99mSTANNOUSPYROPHOSPHATE1+2D2D2F2F2F3+3+3+3+Observer

4+ 4+4+4+4+

CLINICAL SCIENCESDIAGNOSTICNUCLEARMEDICINE

TABLE 2-B. TRUE- AND FALSE-POSITIVEFRACTIONS, TRUE- AND FALSE-NEGATIVE FRACTIONS,AND ACCURACY, AT FIVE CRITERION LEVELS, FOR SIX DIFFERENT OBSERVERS INTERPRETING

123 MYOCARDIAL IMAGES OBTAINED WITH

True-positivefraction

(sensitivity)1

234560.063

0.0420.1790.2320.0420.0320.442

0.3160.747

0.3790.6210895

0.7260.8420.9370.5160.7160.937

0.8420.9050.95806320.9160.947

0.8630.9260.9790.695

0.936True-Negative

Fraction(specificity)1

234561.000

1.0001.0001.0001.0001.0001.000

1.0000.786

0.9640.7860.893

09290.6070.6070.9290.6070.786

0.7500.3930.429

0.8570.3210.607

0.6430.3570.250

0.7860.250False-Positive

Fraction1 234560.000

0.0000.0000.0000.0000.0000.000

0.000

@:2@t@

0.0360.2140.107

0.0710.393

0.3930.0710.3930.214

0.2500.607

0.5710.1430.6790.393

0.3570.643

0.7500.214

0.750False-negative

fraction1 234560.937

0.9580.8210.7680.9580.9680.558

0.684

0.6210.3790.105

0.2740.1580.0630.4840.2840.063

0.1580.0950.0420.3680.0840.053

0.1370.0740.0210.305

0.064Accuracy1

234560.276

0.2600.3660.4070.2600.2520.569

0.472

0.7560.7720.5120.6590.894

0.7720.7890.8620.6100.8910.902

0.821

0.7890.837

0.6830.7800.870

0.8130.7970.813

0.7150.780

. Underlined figures represent the individual observer's optimal criterion level.

Behind the nuclear physicians come both of the cardiologists, with an experience of I8 mo in reading myocardial scintigrams. They gave a moderately good performance. Finally, both fellows, with only a 3-motraining in gamma imaging, showed the poorest performance.

All observersdemonstrated a markedly better execution when interpreting images obtained with PPi thanwith MDP (Table 3). The difference between the bestand worst individual sensitivities obtained with MDP(0.289), was slightly decreasedwhen PPi was used(0.274), while the comparabledifference in specificityobtained with MDP (0.270), was greatly reduced when

PPi was used (0.143).Regarding accuracy, the range of individual perfor

mances was greater with MDP (0.252), than with PPi(0.235). These facts, along with the findings of thetwo-tailed sign test, can only mean that PPi is a bettermyocardial scanning agent than MDP, with a P value of0.05,and that the experienceand visualperceptionof theobserver affect his interpreting performance morestrongly when MDP is the radiotracer. This effect ismore evident in the performance of the less-experiencedobservers.

The higher sensitivity attained with PPi by the sixobservers may be explained by a more efficient uptake

Volume 21, Number 1 5

Page 6: InterobserverVariabilityintheInterpretationofMyocardialImageswithIc …jnm.snmjournals.org/content/21/1/1.full.pdf · 2006. 12. 19. · TABLEFOREACH1-A.ANALYSISOFRESULTS CONSECUTIVEMYOCARDIALSCANSOBSERVERIN127

(UARON. ACERO, CARDENAS. HUERTA. RODRIGUEZ, AND DEGARAY

.99 TRUENEGATIVEFRACTION(spicificity)

_I.o 0.R 0.6 04 02 0 0

.2

.,,r(aI,,

.4“I0

4I―

.6 ‘i

C,-40z

.95

.9

.8

.7

.6

.5

.4.3.2 -

.05

z0I.C.,

IL@@

Id>.@

—0I—c

w

I-

.8

.82

0.01@ @5 .1 23.4.5.ô.TS

FALSE -POSITIVE FRACTION0

2 4 5 5FALSE-POSITIVE FRACTION

1@

(I)I,,

z0

-44P1

.,,C,-4

I

. .2.34

FALSE-POSITIVE FRACTIONFALSE POSITIVE FRACTION

z0I-U4

(Ci.:>>

Cl)c0@@@C1)

(Ci

I-

FIG. 1. Double-probabIlityplotsandreceiveroperatingcharacteristic(ROC)curvesforeachobserver,eitherwithTc-99mdiphosphonateor Tc-99m pyrophosphate.(A) Nuclear-medicinephysicians:Observer1: 16 yr experiencein nuclearmedicineimaging.Observer2:4 yr traininginnuclearmedicineimaging.(B)Cardiologists:Observer3: 20 yr experienceincardiology;Observer4: 4 yrexperienceincardiology;both,with 18 motraininginnuclearmedicineimaging.(C)Fellowsinnuclearmedicine:Observers5 and6: both,withonly3 motrainingin nuclearmedicine.

of this tracer by the damaged cells of the myocardium.In fivepatients,Singhet al. (16) observeda higheruptake of PPi than MDP at the site of the AM!. Similarily,Davis et al. (/7) showed that the concentration of PPiby an experimental AMI in the rat is 2.5 times that ofMDP. This could be explained by the faster bloodclearance of MDP, due to a more rapid bone uptake andfaster renal elimination of this agent, which rapidly decreases the availability of MDP to the damaged myocardial cells (18). We conclude that PPi is the bettertracer for AM! imaging and that MDP is better for bone

scanning.Berman et al. (/4) suggest that a myocardial image

with a 2+ intensity is equivocal when it shows a diffusepattern (2D), but that it should be considered positivefor AMI when its pattern is focal (2F). That is to say,these authors used a fixed criterion level (2F) for callingan image “positive.―Even though we use the same entenia, we have found that Observers 1, 3, and 6, locatedtheir optimal individual criterion level at the thresholdimplied by a 2F grade when using MDP (Table 2-A).Observers 2 and 5 demonstrated a lower optimal mdi

6 THE JOURNAL OF NUCLEAR MEDICINE

ijl NUCLEARPHYSICIANSk -@ IPYPo.—oc:D

Obssrv•r I .(. LMDP.—.@@pyp a.---.ec3

Obs.rvsr 2 4LMDP è----4@J

. , I I I I

z0I-U4

@-; .6

>@p...

a--- 4

p..

Page 7: InterobserverVariabilityintheInterpretationofMyocardialImageswithIc …jnm.snmjournals.org/content/21/1/1.full.pdf · 2006. 12. 19. · TABLEFOREACH1-A.ANALYSISOFRESULTS CONSECUTIVEMYOCARDIALSCANSOBSERVERIN127

Sensitivity Specificity AccuracyObserver MOP PPI MOP PPi MDP PPI

CLINICAL SCIENCESDIAGNOSTIC NUCLEAR MEDICINE

TRUE NEGATIVE FRACTIONIspecèfèclty:t

FIG. 1. (continued).

I .0

z0I-C.,4

IL>1

>zI-'i@c

I@j

I-

I—U4

I―

p..;

p..

FALSE POSITIVE FRACTION FALSE -@SlTlVE FRACTION

Were obtained with MDP, and 0.750 when performedwith PPi.

Observer I demonstrated a higher sensitivity with bothradiopharmaceuticals, but failed in detecting 26 casesof AM!, 16 studied with MDP, ten with PPi. Three ofthese false-negative results (two with MDP and one withPPi), could be explained because they derived fromsubendocardial infarcts, and five (three with MDP, twowith PPi), because the studies were performed duringthe first 48 hr after the onset of the acute episode, a penod in which the influx ofcalcium and its tracer towardthe damaged myocardial cells is lower than in the interval between the third and the fifth day ofevolution ofAM! (19,20). The lower influx of the radiophosphateto the infarcted area could explain the reduced sensitivityof the procedure during this period.

Of the remaining I8 false-negativeresults (11 withMDP, seven with PPi), the majority may be explainedby position of the infarcts, as documented by ECG. Six

vidual criterion level (2D), while Observer 4 preferreda stricter level (3+). What seems to be more importantIs that when using PPi, Observers 2, 3, and 6 shifted theiroptimal criterion levels upward, namely to 2F, 3+, and4+, respectively(Table 2-B). This suggeststhe needfora periodic review of the optimal criterion levels of physicians who interpret myocardial images made with radiophosphates, rather than the specification of a fixedcriterion level for all observers.

Sensitivity of radiophosphate imaging is lower for thedetection of subendocardial infarction than fortnansmural infarction. From 250 images included in thisstudy, only ten were obtained in patients with subendocardial infarction, as documented by ECG; six of thesewere studied with MDP, and four with PP1. All observersfailed in the detection of three nontransmural infarcts,two studied with MDP, one with PPi. This means thatin this rather small sample, sensitivity of the six observersfor subendocardiäl infarction was 0.667 when the images

TABLE 3. COMPARISONSBETWEEN MDP AND PPI WITh SIX OBSERVERS, EACH OPERATING ATHIS OPTIMAL CRITERION LEVEL

123456

0.822 ±0.040 0.895 ±0.031 0.865@ 0.056 0.893 ±0.058 0.835 ±0.033 0.894 ±0.0280.644 ±0.050 0.726 ±0.046 0.892 ±0.051 0.929 ±0.049 0.717 1 0.040 0.772 ±0.0380.678 :1:0.049 0.747 ±0.045 0.622 ±0.080 0.786 ±0.078 0.661 ±0.042 0.756 ±0.0390.533 ±0.053 0.737 ±0.045 0.892 ±0.051 0.893 ±0.058 0.638 ±0.043 0.772 ±0.0380.556 ±0.052 0.632 ±0.049 0.757 ±0.071 0.857 ±0.066 0.614 ±0.043 0.683 ±0.0430.556 :1:0.052 0.621 ±0.050 0.649 ±0.078 0.786 ±0.078 0.583 ±0.044 0.659 ±0.043

0.235Maximaldifferences0.2890.2740.2700.1430.252*

Sign test with twotails indicates thatPPI is the bettermyocardial scanningagent (p = 0.05).

Volume 21, Number 1 7

Page 8: InterobserverVariabilityintheInterpretationofMyocardialImageswithIc …jnm.snmjournals.org/content/21/1/1.full.pdf · 2006. 12. 19. · TABLEFOREACH1-A.ANALYSISOFRESULTS CONSECUTIVEMYOCARDIALSCANSOBSERVERIN127

CUARON, ACERO. CARDENAS, HUERTA, RODRIGUEZ. AND DEGARAY

cases (four with MDP, two with PPi) were ofsmall anteroseptal infarcts, and probably they were masked byuptake in the sternum. Six other false-negative results(three with each tracer) were obtained in cases of smallposteroinfenior infarcts. The greater distance betweenthe infarcted area and the camera may explain the failure to detect these lesions.

To explain the six remaining false-negative results(four with MDP, two with PPi), we must look at thepathophysiology of the AMI in relation to radiophosphate. Four of them were extensive anterolaterals andtwo were extensive infarcts of the anterior wall, so theywere not masked by a radioactive bone and they werelocated in a cardiac region near the camera detector.Furthermore, these studies were performed during thethird and fourth days after the onset ofacute chest pain,which is the period of maximal sensitivity of the procedune.

In one case of an extensive anterolateral infarctstudied with MDP, we observed intense tracer uptakeat the site of a recent fracture of the left humerus. Afaster-than-normal blood clearance of the tracer, causedby the increased bone uptake, could be the reason for thelow myocardial uptake. For the five remaining falsenegative results, we can only suggest that perhaps anunusually poor collateral circulation caused the ischemiato be so severe in the peripheral parts of the infarct thatthey were deprived of calcium and its radiotracer(20,21 ). Since these five patients were the most severelyill in our series, we suspect that a lack of evident radiophosphate uptake, during the most favorable period, mayindicate poorcollateral supply,a@dthusan unfavorableprognosis.

Specificity of the procedure depends on the concentration of the radiotracer in the cardiac area due toprocesses other than AMI. The eight false-positive resuIts obtained by Observers I and 5 include two cases ofleft-ventricular aneurysm due to previous infarction(7,8) onepatientwhohada recentdefibrillationmaneuver (8), one case ofcarcinoma ofthe ovary metastaticto myocardium (8), one patient with unstable arteniosclerotic heart disease (9), one case of scleroderma withdiffuse myocardial damage (10), and two subjects withcalcified mitral valves (5,6).

We may conclude that myocardial scintigraphy withTc-99m-labeled phosphate is a useful procedure for thedetection of acute myocardial infarction, and that PPiis a better tracer than MDP for this purpose. A falsenegative result, during the period of maximal sensitivityof the procedure, may indicate a poor prognosis, as itmay signal a deficient collateral coronary circulation.Nevertheless, it may be also caused by an increased boneuptake of the radiophosphate, which can easily be detected by a whole-body scan. Notwithstanding the useof strict criteria for the visual quantitation of the myocardial tracer uptake—by comparing it with its con

centration in bone—the perception of these shades wasdifferent for each observer. We seem to have demonstrated a correlation between the degree of experienceand visual perception of the observer, and the quality ofhis performance as an interpreter of these images. Wealso suspect that uncorrected visual defects could playan important role in the reading of nuclear medicine images. These results justify the setting of special qualitycontrol systems based on the periodic determination ofthe optimal individual criterion level for every physicianin chargeof the readingof thesestudies,insteadof usinga fixed criterion level by which all the observers call animage “positive―for AMI.

REFERENCES

I. PARKEY RW, BONTE FJ, MEYER SL, et aI: A new method forradionuclide imaging ofacute myocardial infarction in humans.Circulation 50:540-546, 1974

2. HOLMAN QL, TANAKA TT, LFSCH M: Evaluation of radiopharmaceuticals for the detection of acute myocardial infarctioninman.Radiology 121:427-430, 1976

3. ZWEIMANFG,HOLMANBL,O'KEEFEA,etal:Selectiveuptakeof99mTc complexesand 67Ga in acutely infarcted myocardium.JNucIMed 16:975-979,1975

4. ABDULLA AM, CANED0 MI, CORTEZ BC, et aI: Detection ofunstable angina by 99mtechnetiumpyrophosphate myocardialscintigraphy. Chest 69:168-173, 1976

5. JENGO JA, MENA I, JOE SH, et al: The significance of calcificvalvular heart diseasein Tc-99m pyrophosphatemyocardial infarction scanning: Radiographic, scintigraphic, and pathologicalcorrelation. J Nuci Med I 8:776-78 1, 1977

6. EPSTEIN DA: Uptake of Tc-99m diphosphonatein a massivelycalcified mitral annulus: Case report. J Nuci Med 18:799-800,1977

7. AHMADM, DuBIELJP,VERDONTA JRetal:Technctium-99mstannouspyrophosphatemyocardial imaging in patientswith andwithout left ventricular aneurysm. (‘irculation53:833-838,1976

8. CARDENAS M, HUERTA D, DE LA REGUERA GF, et al: Utilidadde Ia centeIIeograf@aconTecnCcio-99mdifosfonatosparael diagnósticodel infartoagudodel miocárdio.Arch Inst Cardiol Mex48:979-994, 1978.

9. WILLERSONJT, PARKEYRW, BONTEFJ,et aI:Technetiumstannouspyrophosphatemyocardial scintigrams in patients withchestpainofvaryingetiology.CirculationSI:1046-1052,I975

10. PEREZ LA, HAYT DB, FREEMAN LM: Localization of myocardial disorders other than infarction with @mTc@IabeIedphosphate agents.J Nuci Med I 7:24 I -246, I 976

II. Go RT, DOTY DB, CHIU CL, et al: New method for diagnosingmyocardial contusion in man by radionuclide imaging. Radiology116:107-110,1975

12. TURNER DA, FORDIIAM EW, PAGANO JV, et a1: Brain scanning with the Anger multiplane tomographic scanner as a secondexamination. Evaluation by the ROC method. Radiology I 21:I15-124,1976

13. TURNER DA, FORDHAM EW, ALl et al: Motion corrected hepatic scintigraphy. An objective clinical evaluation. J NucI Med19:142-148,1978

14. BERMAN DS, AMSTERDAM EA, HINES HM, et a): New approach to interpretation of technetium-99m pyrophosphatescintigraphy in detection ofacute myocardual infarction: Clinicalassessment of diagnostic accuracy. Ant J (‘ardiol39:341 -346,I 977

8 THE JOURNAL OF NUCLEAR MEDICINE

Page 9: InterobserverVariabilityintheInterpretationofMyocardialImageswithIc …jnm.snmjournals.org/content/21/1/1.full.pdf · 2006. 12. 19. · TABLEFOREACH1-A.ANALYSISOFRESULTS CONSECUTIVEMYOCARDIALSCANSOBSERVERIN127

CLINICAL SCIENCESDIAGNOSTICNUCLEARMEDICINE

15. TURNER DA: An intuitive approach to receiver operating characteristic curve analysis.J Nuci Med I 9:2 13—220,1978

16. SINGI-I A, USHER M: Comparison of Tc-99m methylene di. phosphonate with Tc-99m pyrophosphate in the detection of acute

myocardial infarction: Concise communication. J Nucl Med

18:790-792,1977

17. DAVIS MD, HOLMAN BL, CARMEL AN: Evaluation of radiopharmaceuticals sequestered by acutely damaged myocardium.JNuclMed 17:911-917,1976

18. SUBRAMANIAN G, MCAFEE JG, BLAIR RJ, et al: Techne

tium-99m-methylenediphosphonate—asuperioragentforskeletalimaging: Comparison with other technttium complexes.J NucIMed 16:744-755,1975

19. BUJA LM, PARKEY RW, DEEs JH, Ct al: Morphological correlatesoftechnetium-99mstannouspyrophosphateimagingofacutemyocardial infarcts in dogs. Circulation 52:596-607, 1975

20. BUJA LM, PARKEY RW, STOKELY EM, et a1: Pathophysiologyof technetium-99mstannouspyrophosphateand thallium-201scintigraphyof acute anterior myocardial infarcts in dogs.J ClinInvest57:1508-1522,1976

The Computer Council announces its Annual Meeting, to be held conjointly with the meetingsof the SNM BoardofTrustees, the Academic Council, and the RadiopharmaceuticalCouncil. The meeting will feature current researchIn computer technology related to nuclear medicine, specifically techniques utilizing single photon emission tomography.

The Computer Council welcomes the submissionof abstractsfrom membersand non-members of the Society ofNuclear Medicine.Abstractsshould not exceed 200 words.The title, author,and institutionalaffilliations should beincluded at the top of the first page. The name of the author presenting the paper must be underlined. Abstractsshould contain a statement of purpose, the methods used, results,and conclusion.

Original abstractsans supporting data should be sent to:

Ronald R. Price, Ph.D.Department of Radiology and Radiological SciencesVanderbilt University Medical CenterNashville,TN 37232(615)322-2394

Abstracts must be received no later than December 1, 1979.

Volume 21, Number I 9

SNM COMPUTER COUNCILANNUAL MEETING

Miami, FloridaJanuary 27, 1980 Fontainbleu Hotel

ANNOUNCEMENT AND FIRST CALL FOR ABSTRACTS

ANNOUNCEMENTOFBERSON-YALOWAWARD

The Education and ResearchFoundationof the Society of Nuclear Medicine invitesmanuscnptsfor considerationfor the Fourth Annual Berson-Yalow Award. Work will be judged upon originality and contribution to the fields ofbasicorclinicalradioassay.Thewinningauthorwill receiveamonetaryawardof$750.O0andthemanuscriptwillbepresen'ed at the 27th Annual Meeting of the Society of Nuclear Medicine in Detroit, Ml, June 24-27, 1980.

The manuscriptshouldbe approximatelyten pagesin length (typed,double-spaced). A letter requestingconsideration for the award, including the author's full mailing address and telephone number, should accompany the manuscript. Original manuscript and eight copies must be received by February 1, 1980 at the Society of Nuclear Medicineoffice, 475 Park Avenue South, New York, NY 10016, Attn: Mr. Dennis L. Park.

DEADLINEFORRECEIPTOF MANUSCRIPTS:February1, 1980