The validation of Optscore, a software to evaluate cornea photographs Christine M. Toutain-Kidd,...
-
Upload
deonte-edmonson -
Category
Documents
-
view
214 -
download
0
Transcript of The validation of Optscore, a software to evaluate cornea photographs Christine M. Toutain-Kidd,...
The validation of Optscore, The validation of Optscore, a software to evaluate cornea photographsa software to evaluate cornea photographs
Christine M. Toutain-Kidd, Travis Porco, Eric M. Kidd, Thomas M. Lietman, Michael E. Zegans
Dartmouth Medical SchoolHanover, NH - USA
The authors have no financial interests.
Purpose
Digital photo-grading systems for corneal photographs are underdeveloped compared to systems for retinal photographs. Nevertheless these systems could be an important help to clinicians. The Optscore software was created to attempt to fill that void. Its validation study is presented here.
Validation of Optscore, a software to evaluate cornea pictures
Methods
Optscore is based on the Ruby on Rails web technology. One hundred cornea pictures from the Mycotic Ulcer Treatment Trial (MUTT) pilot study (Prajna et al, in press), were looked at by 6 graders (5 medical students plus an ophthalmologist used as Gold Standard). After going through a training module, they evaluated, twice at least 2 weeks apart, several variables commonly used in the diagnosis of a corneal infection: size, location, opacity, neovascularization, conjunctival inflammation and presence of a hypopyon (see list of questions on following slide). The Interclass Correlation Coefficient (ICC) was calculated to assess the intra and inter-grader variability.
Validation of Optscore, a software to evaluate cornea pictures
Question 1:Is the ulcer/scar visible on this picture?1. Yes2. No
Question 2 (if yes in question 1):Circle the lesion on the photo
Question 3 (if yes in question 1):Location of the lesion.1. Entirely in periphery2. Overlaps central 4mm circle and
periphery without filling center3. Entirely in central circle4. Completely fills 4mm circle and
extends into periphery
Question 4 (if yes in question 1):Opacity of the lesion (worst 50% of the
lesion).1. Mild, iris details visible2. Moderate, some iris visible3. Complete, no iris visible
Question 5:Corneal neovascularization (in clock hours).0. 01. 12. 23. 34. 45. 56. 67. 78. 89. 910. 1011. 1112. 12
Question 6:Conjunctival inflammation (in clock hours)0. 01. 12. 23. 34. 45. 56. 67. 78. 89. 910. 1011. 1112. 12
Question 7:Hypopyon.0. No hypopyon1. Hypopyon but non-hemorrhagic2. Hemorrhagic
Question 8: Quality of the picture.1. Good2. Poor but gradable3. Ungradable
Questions asked during the evaluation of all pictures in this study
Validation of Optscore, a software to evaluate cornea pictures
Validation of Optscore, a software to evaluate cornea pictures
Snapshot of the software screen presenting a picture ready to be evaluated (with the 4 mm central circle and the 12 mm outside circle overlapping the limbus) as well as the first question asked to the graders.Once the grader has answered the question, the software automatically goes to the next question.
Snapshot of the screen summarizing the questions and answers provided for a picture. Until the grader clicks on “next image” (not visible on this snapshot), all answers can be revised and changed.
Validation of Optscore, a software to evaluate cornea pictures
The Steroids for Corneal Ulcers Trial (SCUT) and Mycotic Ulcer Treatment Trials (MUTT) I and II are collaborative studies involving the Aravind Eye Hospital in India, the F.I. Proctor Foundation at UCSF and Dartmouth Medical School. We used some of the pictures obtained during the MUTT pilot study in this work.
The following tables present the Interclass Correlation Coefficient (ICC) calculated for all variables, which allows to compare the consistency of each grader and between the graders and the gold standard (GS).
Validation of Optscore, a software to evaluate cornea pictures
Results and Conclusions
Variable Area Location Opacity Neovascularization
Conj. Inflam.
Hypopyon
Overall ICC 0.8942 0.7974 0.6322 0.6313 0.8632 0.8425
Interclass Correlation Coefficient (ICC) calculated for all variables evaluated in this study.
Table A shows the overall ICC for each variable.
A.
Validation of Optscore, a software to evaluate cornea pictures
- The measure of the area and the evaluation of the presence of a hypopyon are the two variables with the highest ICC.- More subjective variables like level of opacity or estimation of neovascularization have a low ICC.
Intergrader variability
Grader A/
GS1
Grader B/
GS1
Grader C/
GS1
Grader D/
GS1
Grader E/
GS1
Area 0.9228 0.9461 0.8966 0.8722 0.9365
Location 0.7877 0.8996 0.8097 0.8496 0.7794
Opacity 0.6509 0.5338 0.6275 0.5885 0.6877
Neovascularization
0.6354 0.6415 0.5161 0.5757 0.6454
Conj. Inflam
0.8722 0.8342 0.8826 0.8287 0.7903
Hypopyon 0.8241 0.8329 0.9416 0.7542 0.8390
B.
Table B presents the intergrader variability calculated between each grader and the Gold Standard’s first grading.
Validation of Optscore, a software to evaluate cornea pictures
- When comparing the evaluations of each grader with the gold standard (GS), the variables with the highest ICC here are also the ones with the highest overall ICC, such as area, location and hypopyon.
- The evaluation of the level conjunctival inflammation also shows in both cases, tables A and B, a high ICC. This variable is continuous but also quite subjective like neovascularization, the subjectivity has to be taken into account. There were many cases where the number of clock hours of inflammation estimated by the graders was different by one or two clock hours (which is not much on the calculation of the ICC but still leads to a lack of consistency).
Validation of Optscore, a software to evaluate cornea pictures
Table C shows the intragrader variability calculated using the two evaluations done by each grader at least 2 weeks apart.
Intragrader variability
GS1/GS2 Grader A Grader B Grader C Grader D Grader E
Area 0.9823 0.9772 0.9836 0.9762 0.9086 0.9771
Location 0.8533 0.7655 0.9481 0.6736 0.7419 0.8132
Opacity 0.6918 0.3915 0.8534 0.6194 0.6402 0.8164
Neovascularization
0.7568 0.8836 0.9026 0.9339 0.8895 0.9019
Conj. Inflam
0.9510 0.9754 0.9793 0.9469 0.9529 0.9364
Hypopyon 0.8879 0.9353 0.8531 0.9416 0.7772 0.8883
C.
- It should be noted that all graders went through the same training module composed of two parts, a power point presentation teaching them about the cornea and the variables that they would have to evaluate, and a training set of pictures to teach them how to use the software. The time required by each grader to perform the evaluations was recorded as well.- Grader D, who overall has a relatively low intragrader ICC, including for non subjective variables such as area size and presence or absence of hypopyon, did the evaluations the fastest.
- This study demonstrates that Optscore can be reliably used to evaluate several variables required for the assessment of corneal infections. - The training of the graders is essential and the importance of spending enough time per picture should be emphasized. - The least subjective variables show the highest correlation amongst graders.
- Optscore is a very flexible software that can be modified to fit any requirements.
- We believe that Optscore could be developed as a diagnostic help and an educational tool.
Validation of Optscore, a software to evaluate cornea pictures