BMJ Open is committed to open peer review. As part of this commitment we make the peer review history of every article we publish publicly available. When an article is published we post the peer reviewers’ comments and the authors’ responses online. We also post the versions of the paper that were used during peer review. These are the versions that the peer review comments apply to. The versions of the paper that follow are the versions that were submitted during the peer review process. They are not the versions of record or the final published versions. They should not be cited or distributed as the published version of this manuscript. BMJ Open is an open access journal and the full, final, typeset and author-corrected version of record of the manuscript is available on our site with no access controls, subscription charges or pay-per-view fees (http://bmjopen.bmj.com). If you have any questions on BMJ Open’s open peer review process please email
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Value of prediction models for risk stratification of
pulmonary subsolid nodules: a retrospective cohort study
Journal: BMJ Open
Manuscript ID bmjopen-2017-019996
Article Type: Research
Date Submitted by the Author: 06-Oct-2017
Complete List of Authors: Kim, Hyungjin; Seoul National University Hospital Park, Chang Min; Department of Radiology Jeon, Sunkyung; Seoul National University Hospital Lee, Jong Hyuk; Seoul National University Hospital Ahn, Su Yeon; Seoul National University Hospital Yoo, Roh-Eul; Seoul National University Hospital Lim, Hyun-ju; National Cancer center Park, Juil; Seoul National University Hospital
Lim, Woo Hyeon; Seoul National University Hospital Hwang, Eui Jin; Seoul National University Hospital Lee, Sang Min; University of Ulsan College of Medicine Goo, Jin Mo; Seoul National University Hospital
<b>Primary Subject Heading</b>:
Oncology
Secondary Subject Heading: Diagnostics
Keywords: subsolid nodule, risk prediction model, Brock model, validation, lung cancer
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open on F
ebruary 5, 2020 by guest. Protected by copyright.
http://bmjopen.bm
j.com/
BM
J Open: first published as 10.1136/bm
jopen-2017-019996 on 24 May 2018. D
ownloaded from
For peer review only
Value of prediction models for risk stratification of
pulmonary subsolid nodules: a retrospective cohort study
Hyungjin Kim1, Chang Min Park
1, 2, 3, Sunkyung Jeon
1, Jong Hyuk Lee
1, Su Yeon Ahn
1, Roh-
Eul Yoo1, Hyun-ju Lim
1, 4, Juil Park
1, Woo Hyeon Lim
1, Eui Jin Hwang
1, Sang Min Lee
5, Jin
Mo Goo1, 2, 3
1Department of Radiology, Seoul National University College of Medicine, Seoul, Korea
2Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul,
Korea
3Seoul National University Cancer Research Institute, Seoul, Korea
4Department of Radiology, National Cancer center, Goyang, Korea
5Department of Radiology and Research Institute of Radiology, University of Ulsan College
of Medicine, Seoul, Korea
* Corresponding author
Chang Min Park, MD, PhD. Department of Radiology, Seoul National University College
of Medicine, 101, Daehak-ro, Jongno-gu, Seoul 110-744, Korea. Tel: 82-2-2072-0367, Fax:
82-2-743-7418. E-mail: [email protected]
Word count: 3134
Page 1 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
ABSTRACT
Objectives: We aimed to validate the performances of two prediction models (Brock and Lee
models) for the differentiation of minimally-invasive and invasive pulmonary
adenocarcinomas (MIA/IPA) from preinvasive lesions among subsolid nodules (SSNs). The
purpose of our study was to evaluate the feasibility of the two models for the risk
stratification of persistent SSNs.
Design: A retrospective cohort study.
Setting: A tertiary university hospital in South Korea.
Participants: 410 patients with 430 SSNs who underwent surgical resection for the
pulmonary adenocarcinoma spectrum.
Primary and secondary outcome measures: Using clinical and radiological variables, the
predicted probability of MIA/IPA was calculated from pre-existing logistic regression models
(Brock and Lee models). Areas under the receiver operating characteristic curve (AUCs)
were calculated and compared between models. Performance metrics including sensitivity,
specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV)
were also obtained.
Results: For pure ground-glass nodules, the AUC of the Brock model in differentiating
MIA/IPA from preinvasive lesions was 0.680. Sensitivity, specificity, accuracy, PPV, and
NPV based on the optimal cutoff value were 62.9%, 68.2%, 65.1%, 73.6%, and 56.6%,
respectively. Sensitivity, specificity, accuracy, PPV, and NPV according to the Lee criteria
were 75.8%, 45.5%, 63.2%, 66.2%, and 57.1%, respectively. For part-solid nodules, the AUC
was 0.748 for the Brock model and 0.769 for the Lee model (P=0.623). Sensitivity, specificity,
Page 2 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
accuracy, PPV, and NPV were 81.5%, 55.6%, 79.3%, 95.3%, and 21.4%, respectively, for the
Brock model and 76.4%, 74.1%, 76.2%, 97.0%, and 22.2%, respectively, for the Lee model.
Conclusions: The performance of prediction models for SSNs in differentiating MIA/IPA
from preinvasive lesions was unsatisfactory. Thus, an alternative risk calculation model
should be developed for incidentally detected SSNs.
Strengths and limitations of this study
• This is the first study to externally validate the performance of pre-existing risk prediction
models for the incidentally detected pulmonary subsolid nodules.
• The main limitation of this study is that it only analyzed surgically resected lung nodules,
thus inducing selection bias.
Page 3 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
INTRODUCTION
Pulmonary subsolid nodules (SSNs) represent a histologic spectrum of
adenocarcinoma and its preinvasive precursors, including atypical adenomatous hyperplasia
(AAH), adenocarcinoma-in-situ (AIS), minimally invasive adenocarcinoma (MIA) and
invasive pulmonary adenocarcinoma (IPA).1 SSNs are common findings at chest computed
tomography (CT) which have been increasingly detected in CT screening studies.2 3
Indeed,
according to one prospective screening study, 4.2% of the participants had at least one pure
ground-glass nodule (pGGN) and 5.0% had at least one part-solid nodule (PSN) at baseline
rounds of screening.2
With this prevalence in mind, numerous studies have justifiably focused on the
differentiation of invasive adenocarcinomas from preinvasive lesions4-13
as invasive
adenocarcinoma requires surgical resection with conventional lobectomy and lymph node
dissection14
whereas preinvasive lesions can be followed-up conservatively with annual CT
surveillance or resected at a lesser extent (sublobar resection).15
Thus, the discrimination of
invasive adenocarcinoma has been a major topic of interest for many radiologists and
clinicians to date.
In a quest to obtain quantitative risk prediction tools for pulmonary nodules,
McWilliams et al.16
developed a prediction model (Brock model) utilizing various clinical
and radiological features. The Brock model demonstrated higher accuracy in determining the
likelihood of malignancy in pulmonary nodules compared to other existing models17
and was
also externally validated in three independent screening populations.18-20
Nevertheless, in the
context that a substantial percentage of persistent SSNs may belong to the adenocarcinoma
spectrum, the performance of the established model in differentiating invasive
Page 4 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
adenocarcinoma should also be validated in order to encourage the use of the model in
routine practice, as suggested by the British Thoracic Society (BTS).21
Lee et al.7 also developed a prediction model (Lee model) using simple size metrics
and morphologic features for the differentiation of invasive adenocarcinomas appearing as
SSNs. The model accuracy was reported to be excellent for the identification of invasive
adenocarcinomas. However, it has also not been tested or validated.
Therefore, we aimed to validate the performances of the two prediction models
(Brock and Lee models) for the differentiation of MIAs and IPAs from preinvasive lesions
among SSNs. The purpose of our study was to evaluate the feasibility of the two models in
the risk stratification of persistent SSNs.
Page 5 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
METHODS
This retrospective analysis was approved by the institutional review board of Seoul
National University Hospital (IRB No. 1705-116-855) and the requirement for written
informed consent was waived.
Study population
We retrospectively reviewed the electronic medical records of our hospital and found
1915 patients who had undergone surgical resection for lung cancer between 2011 and 2015.
Among the 1915 patients, we identified 1073 patients whose pathologic diagnoses belonged
to the pulmonary adenocarcinoma spectrum including AAH, AIS, MIA, and IPA.1 22
Thereafter, we reviewed the thin-section CT images of the patients to include only those with
SSNs (reviewers: J.S.K., J.H.L., S.Y.A., R.E.Y., H.L., and H.K.); 548 patients whose lung
cancers appeared as solid nodules on CT scans were excluded. We also excluded 76 patients
with nodules smaller than 5 mm or larger than 3 cm and 39 patients in whom data regarding
the family history of lung cancer were not available. Consequently, 410 patients were
included in this study. Among these patients, 18 patients had two nodules and one patient had
three nodules. Therefore, a total of 430 nodules from 410 patients were analyzed in the
present study (figure 1). There were 175 men and 235 women (mean age ± standard deviation,
60.9±9.6 years). As for the nodule type, there were 106 pGGNs and 324 PSNs. IPAs were
found in 305 nodules followed by MIA in 54 nodules, AIS in 52 nodules, and AAH in 19
nodules. Median nodule size was 15.6 mm (interquartile range, 11.7, 20.9 mm) (table 1).
Table 1 Demographic data of the entire study population
Characteristics Value
Page 6 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Age (year)a 60.9±9.6
Sex Male 175 (42.7)
Female 235 (57.3)
Pathology AAH 19 (4.4)
AIS 52 (12.1)
MIA 54 (12.6)
IPA 305 (70.9)
Nodule type Pure GGN 106 (24.7)
Part-solid nodule 324 (75.3)
Family history of lung cancer Yes 17 (4.0)
No 413 (96.0)
Emphysema Yes 44 (10.2)
No 386 (89.8)
Nodule size (mm)b 15.6 (11.7, 20.9)
Solid portion size (mm)b 5.7 (1.6, 11.2)
Solid proportion (%) 41.0 (11.4, 62.0)
Location Upper lobe 259 (60.2)
Other lobes 171 (39.8)
Lobulation Yes 143 (33.3)
No 287 (66.7)
Spiculation Yes 140 (32.6)
No 290 (67.4)
Nodule count per scanb 3 (1, 5)
Note.— Unless otherwise specified, data are numbers of patients (with percentages in
parentheses).
aData are mean ± standard deviation.
bData are median (with interquartile range in parentheses).
AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass
nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma
Page 7 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Data collection
Patient characteristics including demographic data were collected from the electronic
medical records of Seoul National University Hospital. Patient age, sex, pathologic diagnosis,
family history of lung cancer, and nodule location (lobe) were recorded. The thin-section CT
images were also reviewed to obtain radiologic information of nodules (nodule type, nodule
size, solid portion size, solid proportion, lobulation, spiculation, and nodule count per scan)
and the background lung parenchyma (presence of visually detected emphysema). These
features were used as input variables for logistic regression analysis at the Brock model16
and
Lee model.7 Nodule size and solid portion size were measured as the maximum transverse
diameter (mm) using an electronic caliper. Solid proportion (%) was calculated as the solid
portion size divided by the nodule size. Nodule count was defined as the total number of non-
calcified nodules at least 1 mm in diameter.16
Image review was conducted by three
radiologists (J.P., W.H.L., and H.K.) and each nodule was analyzed once by one of these
radiologists. Details regarding the CT scanning protocols are described in the online
supplementary material.
Measurement variability
We previously analyzed and reported the measurement variability of SSNs and solid
portion size using two same-day repeat CT scans.23
Measurement variability range for the
maximum transverse diameter of SSNs on lung window CT images was ±2.2 mm. For the
solid portion, it was ±3.7 mm. Inter-reader agreement (κ) of nodule type ranged from 0.80 to
0.96. Therefore, we did not re-evaluate the measurement variability or inter-reader agreement
of nodule type in this study.
Statistical analysis
Page 8 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
To investigate whether the variables incorporated in the established models (Brock
and Lee models) were significantly different between preinvasive (AAH and AIS) and
invasive lesions (MIA and IPA), we first performed univariate analysis. Categorical variables
were analyzed using the Pearson Chi-square test or Fisher’s exact test and continuous
variables were analyzed using the independent t-test or Mann-Whitney U test, as appropriate.
We then calculated the predicted probability from each logistic regression model. For
the Brock model, a full model with spiculation was used with input variables of age, sex,
family history of lung cancer, emphysema, nodule size, nodule type, nodule location, nodule
count per scan, and spiculation.16
Nodule size was subjected to power transformation prior to
entry as described previously.16
Age and nodule count were centered at a mean of 62 years
and 4, respectively.16
We recorded the predicted probability of each nodule, which was a
continuous value from 0 to 1 (0 to 100%). For the Lee model, two different methods were
used for analysis. For pGGNs, a single cutoff of nodule size (≥10 mm) was used to
discriminate invasive lesions as stated by Lee et al.7 In the case of PSNs, four variables
(nodule size, solid proportion, lobulation, and spiculation) were substituted into the
regression formula.7 Therefore, predicted probability was obtained only for PSNs in terms of
the Lee model. No preprocessing of variables was performed.
With the predicted probability obtained through each model, receiver operating
characteristic curve (ROC) analysis was performed to investigate the discriminative
performance of the prediction models in diagnosing invasive lesions. Areas under the ROC
curve (AUCs) were obtained and an optimal cutoff value based on the Youden index was
recorded. We calculated the sensitivity, specificity, accuracy, positive predictive value (PPV),
and negative predictive value (NPV) of each model with the optimal cutoff. For the Brock
model, two different cutoffs were applied to the calculation: 1) a threshold of 10% risk of
Page 9 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
malignancy as suggested by the BTS21
and 2) an optimal cutoff based on the Youden index.
AUCs were compared between the models based on DeLong’s method.24
As the
predicted probability of pGGNs was not available for the Lee model, AUC comparison was
conducted only for PSNs. Diagnostic accuracy was also compared between the models using
the McNemar test. All statistical analyses were performed using two commercial software
programs (MedCalc version 12.3.0, MedCalc Software, Mariakerke, Belgium; and SPSS 19.0,
IBM SPSS Statistics, Armonk, NY, USA). A P-value <0.05 was considered to indicate
statistical significance.
Page 10 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
RESULTS
Pathologic diagnoses of pGGNs and PSNs
Among 106 pGGNs, 44 were preinvasive and 62 were invasive lesions. As for the
324 PSNs, 27 were preinvasive and 297 were invasive lesions.
Comparisons between preinvasive and invasive lesions
For pGGNs, a family history of lung cancer was more frequently observed in patients
with invasive lesions (6/62) than in those with preinvasive lesions (0/44; P=0.040). Invasive
lesions (14.1±5.3 mm) were also significantly larger than preinvasive lesions (10.8±4.1 mm;
P=0.001). In addition, patients with invasive lesions had a smaller nodule count per scan
(invasive vs. preinvasive lesions: median, 2 vs. 4 nodules per scan; P=0.008). There were no
significant differences in age, sex, presence of emphysema, nodule location, lobulation, and
spiculation (table 2).
Table 2 Comparison of clinical and radiological characteristics between differing pathologic
diagnoses in patients with pure GGNs
Characteristics AAH and AIS
(n=44)
MIA and IPA
(n=62)
P-value
Age (year)a 56.2±10.3 57.8±10.4 0.434
Sex Male 18 (40.9) 31 (50.0) 0.430
Female 26 (59.1) 31 (50.0)
Family history of lung
cancer
Yes 0 (0) 6 (9.7) 0.040
No 44 (100) 56 (90.3)
Emphysema Yes 5 (11.4) 5 (8.1) 0.738
Page 11 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
No 39 (88.6) 57 (91.9)
Nodule size (mm)a 10.8±4.1 14.1±5.3 0.001
Location Upper lobe 26 (59.1) 33 (53.2) 0.560
Other lobes 18 (40.9) 29 (46.8)
Lobulation Yes 3 (6.8) 12 (19.4) 0.091
No 41 (93.2) 50 (80.6)
Spiculation Yes 1 (2.3) 7 (11.3) 0.136
No 43 (97.7) 55 (88.7)
Nodule count per scanb 4 (2, 7) 2 (1, 4) 0.008
Note.— Unless otherwise specified, data are numbers of patients (with percentages in
parentheses).
aData are mean ± standard deviation.
bData are median (with interquartile range in parentheses).
AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass
nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma
For PSNs, nodule size, solid portion size, and solid proportion were significantly
larger in invasive lesions (invasive vs. preinvasive lesions: mean nodule size, 18.1±5.8 mm vs.
13.5±4.9 mm, P<0.001; median solid portion size, 8.4 mm vs. 4.7 mm, P<0.001; mean solid
proportion, 51.6±20.3% vs. 42.2±18.6%, P=0.018). Lobulation and spiculation were more
frequently observed in invasive lesions (invasive vs. preinvasive lesions: lobulation, 123/297
vs. 5/27, P=0.023; spiculation, 128/297 vs. 4/27, P=0.004). There were no significant
differences in age, sex, family history of lung cancer, presence of emphysema, nodule
location, and nodule count per scan (table 3).
Table 3 Comparison of clinical and radiological characteristics between differing pathologic
diagnoses in patients with part-solid nodules
Page 12 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Note.— Unless otherwise specified, data are numbers of patients (with percentages in
parentheses).
aData are mean ± standard deviation.
bData are median (with interquartile range in parentheses).
AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass
nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma
SSN risk stratification using the Brock and Lee models
Characteristics AAH and AIS
(n=27)
MIA and IPA
(n=297)
P-value
Age (year)a 59.6±9.4 62.6±8.9 0.125
Sex Male 7 (25.9) 127 (42.8) 0.104
Female 20 (74.1) 170 (57.2)
Family history of lung
cancer
Yes 1 (3.7) 10 (3.4) 1.000
No 26 (96.3) 287 (96.6)
Emphysema Yes 2 (7.4) 32 (10.8) 0.753
No 25 (92.6) 265 (89.2)
Nodule size (mm)a 13.5±4.9 18.1±5.8 <0.001
Solid portion size
(mm)b
4.7 (3.5, 5.5) 8.4 (5.1, 13.6) <0.001
Solid proportion (%)a 42.2±18.6 51.6±20.3 0.018
Location Upper lobe 12 (44.4) 188 (63.3) 0.064
Other lobes 15 (55.6) 109 (36.7)
Lobulation Yes 5 (18.5) 123 (41.4) 0.023
No 22 (81.5) 174 (58.6)
Spiculation Yes 4 (14.8) 128 (43.1) 0.004
No 23 (85.2) 169 (56.9)
Nodule count per scanb 3 (2,8) 3 (1,5) 0.207
Page 13 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
For pGGNs, the AUC of the Brock model’s predicted probability for differentiating
invasive lesions from preinvasive lesions was 0.680 [95% confidence interval (CI): 0.582,
0.767] (table 4). A cutoff of 10%, suggested by the BTS, yielded a sensitivity, specificity,
accuracy, PPV, and NPV of 32.3%, 90.9%, 56.6%, 83.3%, and 48.8%, respectively. Another
cutoff of 4.29%, an optimal threshold based on the Youden index, provided a sensitivity,
specificity, accuracy, PPV, and NPV of 62.9%, 68.2%, 65.1%, 73.6%, and 56.6%. A nodule
size cutoff (10 mm) suggested by Lee et al.7 was also applied to our study population. The
resultant sensitivity, specificity, accuracy, PPV, and NPV were 75.8%, 45.5%, 63.2%, 66.2%,
and 57.1%, respectively. There were no significant differences in diagnostic accuracy
between the Brock model and Lee criteria (Brock model cutoff 10% vs. nodule size cutoff
10mm, P=0.382; Brock model cutoff 4.29% vs. nodule size cutoff 10 mm, P=0.832).
Table 4 Performance of each prediction model in diagnosing minimally invasive
adenocarcinoma and invasive adenocarcinoma for pure ground-glass nodules
Note.— Data are in percentages except for AUC.
aA cutoff of 10% predicted probability was used as suggested by the British Thoracic Society.
bA cutoff of 4.29% predicted probability, the optimal threshold based on the Youden index,
was used.
cA cutoff of 10 mm nodule size was used.
AUC, area under the receiver operating characteristic curve; NPV, negative predictive value;
Sensitivity Specificity Accuracy PPV NPV AUC
Brock model1a 32.3 90.9 56.6 83.3 48.8 0.680
(0.582, 0.767) Brock model2b 62.9 68.2 65.1 73.6 56.6
Lee model
(Nodule sizec)
75.8 45.5 63.2 66.2 57.1 -
Page 14 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
PPV, positive predictive value
As for PSNs, the AUC of the Brock model was 0.748 (95% CI: 0.697, 0.795) for the
discrimination of invasive lesions from preinvasive lesions (table 5). A cutoff of 10% yielded
a sensitivity, specificity, accuracy, PPV, and NPV of 81.5%, 55.6%, 79.3%, 95.3%, and
21.4%. The optimal cutoff based on the Youden index was 10.11%, which was very close to
the suggested threshold by the BTS. Therefore, performance metrics were not calculated
separately. AUC of the Lee model was 0.769 (95% CI: 0.720, 0.814), and an optimal cutoff of
66.68% provided a sensitivity, specificity, accuracy, PPV, and NPV of 76.4%, 74.1%, 76.2%,
97.0%, and 22.2%, respectively. AUCs and diagnostic accuracies were not significantly
different between the two models (P=0.623 and 0.237, respectively).
Table 5 Performance of each prediction model in diagnosing minimally invasive
adenocarcinoma and invasive adenocarcinoma for part-solid nodules
Note.— Data are in percentages except for AUC.
aA cutoff of 10% predicted probability was used, which was suggested by the British
Thoracic Society and was optimal at the same time.
bAn optimal cutoff of 66.68% was adopted based on the Youden index.
AUC, area under the receiver operating characteristic curve; NPV, negative predictive value;
PPV, positive predictive value
Sensitivity Specificity Accuracy PPV NPV AUC
Brock modela 81.5 55.6 79.3 95.3 21.4 0.748
(0.697, 0.795)
Lee modelb 76.4 74.1 76.2 97.0 22.2 0.769
(0.720, 0.814)
Page 15 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
DISCUSSION
In this study, we revealed that AUCs for the differentiation of invasive lesions among
PSNs using the established risk prediction models ranged from 0.748 to 0.769 with no
significant differences between the two models. Diagnostic accuracies based on the optimal
cutoffs were 79.3% for the Brock model and 76.2% for the Lee model. For pGGNs, the
diagnostic accuracy was 56.6%-65.1% for the Brock model depending on the cutoff values
used and 63.2% for the Lee criteria.
McWilliams et al.16
originally developed a lung cancer prediction model (Brock
model) using participants enrolled in a lung cancer screening study and chemoprevention
trials. Thus, the Brock model initially targeted pulmonary nodules detected on first screening
CT. Incidentally detected nodules as well as surgical candidates were not the original target
lesions of this model. However, at present, the BTS recommends using the same diagnostic
approach for nodules detected incidentally as those detected through screening.21
BTS also
recommends using the Brock model for the risk calculation of both solid and subsolid
nodules.21
A cutoff of 10% predicted probability for malignancy is suggested in order to
differentiate high risk SSNs for the performance of biopsy or surgical resection.21
This
quantitative diagnostic approach is to discern malignant SSNs with an appropriate false
positive rate. However, it must be noted that most persistent SSNs belong to one of the four
categories of the adenocarcinoma spectrum: AAH, AIS, MIA and IPA. Therefore, the
potential of the risk prediction model in discriminating lesions with invasive components
(MIA and IPA) should also be tested using pathologic diagnosis as a reference standard.
Indeed, as clinical management strategies differ substantially between preinvasive and
invasive lesions, if it can be feasible to predict the invasiveness of SSNs, clinical planning
Page 16 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
whether to perform annual CT surveillance, limited resection, or conventional lobectomy can
be facilitated.
The performance of the prediction models for the risk stratification of SSNs was not
optimal according to our study results. For PSNs, AUCs of the two models ranged between
0.748 and 0.769 with diagnostic accuracies close to 80%. Past research on distinguishing
invasive adenocarcinomas appearing as SSNs have reported that logistic regression models
built with size metrics, morphologic features, or texture features showed AUCs ranging from
0.79 to 0.98.4-7 9 11 13
However, these models were not tested for an independent cohort or
validated externally. Therefore, their true performance may be lower than that reported in the
literature.
Another important finding of our study was that the PPV for the differentiation of
invasive lesions among PSNs was very high for both models, over 95%. In other words, the
probability of being an invasive lesion was over 95% for nodules predicted as being invasive
through these models. A concern, however, is the high false negative rate of these models.
PSNs predicted as preinvasive lesions, which have a low calculated risk, should be managed
according to their solid portion size.25
PSNs with solid components <6 mm require
surveillance CT annually, while those with solid components ≥6 mm should be biopsied or
surgically resected in consideration of invasive adenocarcinoma.
The diagnostic accuracies of both the Brock model and Lee criteria were even lower
for pGGNs. Among multiple clinical and radiologic characteristics investigated in our study,
only three variables (family history of lung cancer, nodule size, and nodule count per scan)
were significantly different between preinvasive and invasive lesions. This implies the need
for other useful features for the development of new better prediction models. Features such
as nodule volume, mass, or radiomic features may provide additional clues for their
Page 17 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
differentiation.4 26
In addition, changes in nodule characteristics at follow-up CT scans, such
as an increase in nodule size, attenuation, or new development of a solid portion, may also be
valuable for the discrimination.14
Alternatively, computational classification analysis
including deep learning algorithms, which do not require hand-crafted features and can be
self-trained directly from raw image pixels, may be another solution for the diagnosis of
pGGNs.27
The Brock model has been externally validated for the cohorts of the Danish Lung
Cancer Screening Trial (DLCST)19
and National Lung Screening Trial (NLST);18
AUCs for
the discrimination of malignant from benign nodules ranged from 0.834 for the former and
0.963 for the latter. AUCs for the validation cohort of the original paper, the British Columbia
Cancer Agency (BCCA) chemoprevention trial cohort, was 0.970.16
In addition, for an
Australian lung cancer screening cohort, Zhao et al.20
tested the utility of the Brock model for
the baseline evaluation of 52 SSNs and demonstrated that the AUC was 0.89. To the contrary,
however, the model performance evaluated in our study was lower than those reported in the
literature. The main reason for such a discrepancy may be that we included patients who
underwent surgical resection of SSNs unlike previous studies. Thus, the proportion of
preinvasive lesions was small (16.5%) and a major portion of our study population consisted
of invasive lesions (83.5%). Such high prevalence of invasive lesions would have affected
our study results. Nevertheless, SSNs of interest in daily clinical practice may be closer to
those in our study. In routine practice, transient SSNs, which are definitely benign, do not
require risk calculation as they are easily confirmed through follow-up CT scans at short-term
intervals.25
In addition, small SSNs <6 mm are usually preinvasive and do not require CT
surveillance. On the other hand, particular concern should be given to persistent SSNs ≥6 mm,
especially to those with solid components. As the role of biopsy or positron emission
Page 18 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
tomography is limited for SSNs,15
we supposed that risk prediction models may provide
value for more appropriate management planning. In this context, we applied prediction
models to surgically resected SSNs for the validation of their clinical utility.
There were several limitations to our study. First, our study was not conducted for a
screening cohort as described earlier in this manuscript. Second, optimal cutoffs for the
models were not obtained from ROC analyses of the original study populations from which
the models were derived. Therefore, performance metrics including sensitivity, specificity,
accuracy, PPV and NPV may have been overestimated in our study. Third, radiologic nodule
information was extracted from our heterogeneous CT dataset, in which CT acquisition
parameters such as radiation dosage, slice thickness or contrast-enhancement were not
uniform across the study population. However, these factors would have had little effect on
the variables we used. In addition, all CT scans had thin-section images (slice thickness ≤1.5
mm). Fourth, nodule size as well as solid portion size were measured as the longest
transverse diameter in accordance with the definition of lesion size and solid proportion in the
original papers. However, recent analyses have revealed that the usage of average diameter as
an input variable may enhance the model performance.28 29
In conclusion, the performance of the Brock model and Lee model for the
differentiation of invasive lesions among SSNs was suboptimal. In particular, both models
showed lower performance for pGGNs compared to that for PSNs. Thus, an alternative
approach such as computer-aided classification should be developed for the pre-operative
diagnosis of invasive lesions among SSNs.
Page 19 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Acknowledgements
We would like to thank Chris Woo, B.A., for editorial assistance.
Contributors
HK and CMP contributed to conception and design; HK, SJ, JHL, SYA, REY, HL, JP,
WHL, EJH, SML and JMG contributed to acquisition of data, or analysis and interpretation
of data; HK, CMP and JMG were involved in drafting the manuscript or revising it critically
for important intellectual content; all authors gave approval of the final version of the
manuscript.
Funding
This study was supported by Basic Science Research Program through the National
Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT & Future
Planning (grant number: 2017R1A2B4008517).
Competing interests
None declared.
Patient consent
The requirement for written informed consent was waived by the Institutional
Review Board.
Ethics approval
Institutional Review Board of Seoul National University Hospital.
Page 20 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Data sharing statement
All data are available from the corresponding author.
Page 21 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
REFERENCES
1. Austin JH, Garg K, Aberle D, et al. Radiologic implications of the 2011 classification of
adenocarcinoma of the lung. Radiology 2013;266:62-71.
2. Yankelevitz DF, Yip R, Smith JP, et al. CT screening for lung cancer: nonsolid nodules in
baseline and annual repeat rounds. Radiology 2015;277:555-64.
3. Scholten ET, de Jong PA, de Hoop B, et al. Towards a close computed tomography
monitoring approach for screen detected subsolid pulmonary nodules? Eur Respir J
2015;45:765-73.
4. Chae HD, Park CM, Park SJ, et al. Computerized texture analysis of persistent part-solid
ground-glass nodules: differentiation of preinvasive lesions from invasive pulmonary
adenocarcinomas. Radiology 2014;273:285-93.
5. Ding H, Shi J, Zhou X, et al. Value of CT characteristics in predicting invasiveness of
adenocarcinoma presented as pulmonary ground-glass nodules. Thorac Cardiovasc
Surg 2017;65:136-41.
6. Jin C, Cao J, Cai Y, et al. A nomogram for predicting the risk of invasive pulmonary
adenocarcinoma for patients with solitary peripheral subsolid nodules. J Thorac
Cardiovasc Surg 2017;153:462-69 e1.
7. Lee SM, Park CM, Goo JM, et al. Invasive pulmonary adenocarcinomas versus preinvasive
lesions appearing as ground-glass nodules: differentiation by using CT features.
Radiology 2013;268:265-73.
8. Li Q, Fan L, Cao ET, et al. Quantitative CT analysis of pulmonary pure ground-glass
nodule predicts histological invasiveness. Eur J Radiol 2017;89:67-71.
9. Liang J, Xu XQ, Xu H, et al. Using the CT features to differentiate invasive pulmonary
adenocarcinoma from pre-invasive lesion appearing as pure or mixed ground-glass
Page 22 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
nodules. Br J Radiol 2015;88:20140811.
10. Moon Y, Sung SW, Lee KY, et al. Pure ground-glass opacity on chest computed
tomography: predictive factors for invasive adenocarcinoma. J Thorac Dis
2016;8:1561-70.
11. Son JY, Lee HY, Kim JH, et al. Quantitative CT analysis of pulmonary ground-glass
opacity nodules for distinguishing invasive adenocarcinoma from non-invasive or
minimally invasive adenocarcinoma: the added value of using iodine mapping. Eur
Radiol 2016;26:43-54.
12. Yanagawa M, Johkoh T, Noguchi M, et al. Radiological prediction of tumor invasiveness
of lung adenocarcinoma on thin-section CT. Medicine (Baltimore) 2017;96:e6331.
13. Zhang Y, Shen Y, Qiang JW, et al. HRCT features distinguishing pre-invasive from
invasive pulmonary adenocarcinomas appearing as ground-glass nodules. Eur Radiol
2016;26:2921-8.
14. Kim H, Park CM, Koh JM, et al. Pulmonary subsolid nodules: what radiologists need to
know about the imaging features and management strategy. Diagn Interv Radiol
2014;20:47-57.
15. Naidich DP, Bankier AA, MacMahon H, et al. Recommendations for the management of
subsolid pulmonary nodules detected at CT: a statement from the Fleischner society.
Radiology 2013;266:304-17.
16. McWilliams A, Tammemagi MC, Mayo JR, et al. Probability of cancer in pulmonary
nodules detected on first screening CT. N Engl J Med 2013;369:910-9.
17. Al-Ameri A, Malhotra P, Thygesen H, et al. Risk of malignancy in pulmonary nodules: a
validation study of four prediction models. Lung Cancer 2015;89:27-30.
18. White CS, Dharaiya E, Campbell E, et al. The Vancouver lung cancer risk prediction
model: assessment by using a subset of the national lung screening trial cohort.
Page 23 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Radiology 2017;283:264-72.
19. Winkler Wille MM, van Riel SJ, Saghir Z, et al. Predictive accuracy of the PanCan lung
cancer risk prediction model -external validation based on CT from the Danish lung
cancer screening trial. Eur Radiol 2015;25:3093-9.
20. Zhao H, Marshall HM, Yang IA, et al. Screen-detected subsolid pulmonary nodules: long-
term follow-up and application of the PanCan lung cancer risk prediction model. Br J
Radiol 2016;89:20160016.
21. Callister ME, Baldwin DR, Akram AR, et al. British Thoracic Society guidelines for the
investigation and management of pulmonary nodules. Thorax 2015;70 Suppl 2:ii1-
ii54.
22. Travis WD, Brambilla E, Noguchi M, et al. International association for the study of lung
cancer/American thoracic society/European respiratory society international
multidisciplinary classification of lung adenocarcinoma. J Thorac Oncol 2011;6:244-
85.
23. Kim H, Park CM, Song YS, et al. Measurement variability of persistent pulmonary
subsolid nodules on same-day repeat CT: what is the threshold to determine true
nodule growth during follow-up? PLoS One 2016;11:e0148853.
24. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more
correlated receiver operating characteristic curves: a nonparametric approach.
Biometrics 1988;44:837-45.
25. MacMahon H, Naidich DP, Goo JM, et al. Guidelines for management of incidental
pulmonary nodules detected on CT images: from the Fleischner society 2017.
Radiology 2017;284:228-43.
26. Song YS, Park CM, Park SJ, et al. Volume and mass doubling times of persistent
pulmonary subsolid nodules detected in patients without known malignancy.
Page 24 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Radiology 2014;273:276-84.
27. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer
with deep neural networks. Nature 2017;542:115-18.
28. Kovalchik SA, Tammemagi M, Berg CD, et al. Targeting of low-dose CT screening
according to the risk of lung-cancer death. N Engl J Med 2013;369:245-54.
29. van Riel SJ, Ciompi F, Jacobs C, et al. Malignancy risk estimation of screen-detected
nodules at baseline CT: comparison of the PanCan model, Lung-RADS and NCCN
guidelines. Eur RadiolPublished Online First: 14 March 2017. doi:10.1007/s00330-
017-4767-2
Page 25 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
FIGURE LEGENDS
Figure 1 Flow chart of patient inclusion and exclusion.
pGGN, pure ground-glass nodule; PSN, part-solid nodule; SSN, subsolid nodule
Page 26 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Flow chart of patient inclusion and exclusion.
127x158mm (600 x 600 DPI)
Page 27 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
CT SCANNING PROTOCOLS FOR THE STUDY POPULATION
Patients underwent CT scans using seven different scanners from four vendors. CT scans in our study were performed with the
following scanning protocols.
Vendor Toshibaa Siemens
b Siemens
b GE
c GE
c Philips
d Philips
d
Machine Aquilion One Definition Sensation
16
LightSpeed
Ultra
Discovery
CT750 HD
Brilliance 64 Ingenuity
kVp 120 120 120 120 120 120 120
mAs Target standard
deviation of
noise, 14
Quality
reference
mAs, 150
Quality
reference
mAs, 100
Noise index,
16.36
Noise
index, 22.67
Reference
mAs, 200
Reference
mAs, 170
Automatic
exposure
control
Sure Exposure CARE
Dose 4D
CARE
Dose 4D
Auto mA Auto mA Doseright
ACS, Z
DOM
Doseright
ACS, Z
DOM
Rotation time
(sec)
0.5 0.5 0.5 0.5 0.5 0.5 0.5
Collimation
(mm)
0.5x100 0.6x64 0.75x16 1.25x8 0.625x64 0.625x64 0.625x64
Slice thickness
(mm)
1 1 1 1 1.25 1 1
Pitch HP 103 1 1 0.875 0.984 1.014 1.0015
Page 28 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Kernel FC 15-H B60f B60f Bone Chest YC 0 YC 0
Increment 1 1 1 1 1.25 1 1
Matrix 512x512 512x512 512x512 512x512 512x512 512x512 512x512
* All patients were scanned in supine position from the lung apex to the base during suspended maximum inspiration.
* For contrast enhancement, a total of 70-90 mL of 370 mgI/mL of the contrast material, iopamidol (Pamiray 370; Dongkook Pharmaceutical,
Seoul, Korea) or iopromide (Ultravist 370; Schering, Berlin, Germany), was injected at a rate of 2.0-2.3mL/sec using a power injector. CT
scanning was performed with a 60-second delay.
aToshiba Medical Systems, Otawara, Japan
bSiemens Healthcare, Forchheim, Germany
cGE Healthcare, Waukesha, WI, USA
dPhilips Healthcare, Cleveland, OH, USA
Page 29 of 29
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Validation of prediction models for risk stratification of
incidentally detected pulmonary subsolid nodules: a
retrospective cohort study in a Korean tertiary medical
center
Journal: BMJ Open
Manuscript ID bmjopen-2017-019996.R1
Article Type: Research
Date Submitted by the Author: 03-Jan-2018
Complete List of Authors: Kim, Hyungjin; Seoul National University Hospital Park, Chang Min; Department of Radiology Jeon, Sun Kyung; Seoul National University Hospital Lee, Jong Hyuk; Seoul National University Hospital Ahn, Su Yeon; Seoul National University Hospital Yoo, Roh-Eul; Seoul National University Hospital Lim, Hyun-ju; National Cancer center Park, Juil; Seoul National University Hospital Lim, Woo Hyeon; Seoul National University Hospital Hwang, Eui Jin; Seoul National University Hospital
Lee, Sang Min; University of Ulsan College of Medicine Goo, Jin Mo; Seoul National University Hospital
<b>Primary Subject Heading</b>:
Oncology
Secondary Subject Heading: Diagnostics
Keywords: subsolid nodule, risk prediction model, Brock model, validation, lung cancer
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open on F
ebruary 5, 2020 by guest. Protected by copyright.
http://bmjopen.bm
j.com/
BM
J Open: first published as 10.1136/bm
jopen-2017-019996 on 24 May 2018. D
ownloaded from
For peer review only
Validation of prediction models for risk stratification of
incidentally detected pulmonary subsolid nodules: a
retrospective cohort study in a Korean tertiary medical
center
Hyungjin Kim1, Chang Min Park
1, 2, 3, Sunkyung Jeon
1, Jong Hyuk Lee
1, Su Yeon Ahn
1, Roh-
Eul Yoo1, Hyun-ju Lim
1, 4, Juil Park
1, Woo Hyeon Lim
1, Eui Jin Hwang
1, Sang Min Lee
5, Jin
Mo Goo1, 2, 3
1Department of Radiology, Seoul National University College of Medicine, Seoul, Korea
2Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul,
Korea
3Seoul National University Cancer Research Institute, Seoul, Korea
4Department of Radiology, National Cancer center, Goyang, Korea
5Department of Radiology and Research Institute of Radiology, University of Ulsan College
of Medicine, Seoul, Korea
* Corresponding author
Chang Min Park, MD, PhD. Department of Radiology, Seoul National University College
of Medicine, 101, Daehak-ro, Jongno-gu, Seoul 110-744, Korea. Tel: 82-2-2072-0367, Fax:
Page 1 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
82-2-743-7418. E-mail: [email protected]
Word count: 3281
Page 2 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
ABSTRACT
Objectives: We aimed to validate the performances of two prediction models (Brock and Lee
models) for the differentiation of minimally-invasive and invasive pulmonary
adenocarcinomas (MIA/IPA) from preinvasive lesions among subsolid nodules (SSNs). The
purpose of our study was to evaluate the feasibility of the two models for the risk
stratification of persistent SSNs.
Design: A retrospective cohort study.
Setting: A tertiary university hospital in South Korea.
Participants: 410 patients with 410 incidentally detected SSNs who underwent surgical
resection for the pulmonary adenocarcinoma spectrum between 2011 and 2015.
Primary and secondary outcome measures: Using clinical and radiological variables, the
predicted probability of MIA/IPA was calculated from pre-existing logistic regression models
(Brock and Lee models). Areas under the receiver operating characteristic curve (AUCs)
were calculated and compared between models. Performance metrics including sensitivity,
specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV)
were also obtained.
Results: For pure ground-glass nodules (n=101), the AUC of the Brock model in
differentiating MIA/IPA (59/101) from preinvasive lesions (42/101) was 0.671. Sensitivity,
specificity, accuracy, PPV, and NPV based on the optimal cutoff value were 64.4%, 64.3%,
64.4%, 71.7%, and 56.3%, respectively. Sensitivity, specificity, accuracy, PPV, and NPV
according to the Lee criteria were 76.3%, 42.9%, 62.4%, 65.2%, and 56.3%, respectively. For
part-solid nodules (n=309; 26 preinvasive lesions and 283 MIA /IPAs), the AUC was 0.746
Page 3 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
for the Brock model and 0.771 for the Lee model (P=0.574). Sensitivity, specificity, accuracy,
PPV, and NPV were 82.3%, 53.8%, 79.9%, 95.1%, and 21.9%, respectively, for the Brock
model and 77.0%, 69.2%, 76.4%, 96.5%, and 21.7%, respectively, for the Lee model.
Conclusions: The performance of prediction models for the incidentally detected SSNs in
differentiating MIA/IPA from preinvasive lesions might be suboptimal. Thus, an alternative
risk calculation model is required for the incidentally detected SSNs.
Strengths and limitations of this study
� This is the first study to externally validate the performance of pre-existing risk prediction
models for the incidentally detected pulmonary subsolid nodules.
� This study performed head-to-head comparisons between the prediction models for the risk
stratification of subsolid nodules.
� The main limitation of this study is that it only analyzed surgically resected lung nodules,
thus inducing selection bias.
� Study population was small to conduct separate analyses for the pure ground-glass nodules
and part-solid nodules.
Page 4 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
INTRODUCTION
Pulmonary subsolid nodules (SSNs) represent a histologic spectrum of
adenocarcinoma and its preinvasive precursors, including atypical adenomatous hyperplasia
(AAH) and adenocarcinoma-in-situ (AIS).1 SSNs are common findings at chest computed
tomography (CT) which have been increasingly detected in CT screening studies.2 3
Indeed,
according to one prospective screening study, 4.2% of the participants had at least one pure
ground-glass nodule (pGGN) and 5.0% had at least one part-solid nodule (PSN) at baseline
rounds of screening.2
With this prevalence in mind, numerous studies have justifiably focused on the
differentiation of invasive adenocarcinomas from preinvasive lesions4-13
as invasive
adenocarcinoma requires surgical resection with conventional lobectomy and lymph node
dissection14
whereas preinvasive lesions can be followed-up conservatively with annual CT
surveillance or resected at a lesser extent (sublobar resection).15
Thus, the discrimination of
invasive adenocarcinoma has been a major topic of interest for many radiologists and
clinicians to date.
In a quest to obtain quantitative risk prediction tools for pulmonary nodules,
McWilliams et al.16
developed a prediction model (Brock model) utilizing various clinical
and radiological features. The Brock model demonstrated higher accuracy in determining the
likelihood of malignancy in pulmonary nodules compared to other existing models17
and was
also externally validated in three independent screening populations.18-20
Nevertheless, in the
context that a substantial percentage of persistent SSNs may belong to the adenocarcinoma
spectrum, the performance of the established model in differentiating invasive
Page 5 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
adenocarcinoma should also be validated in order to encourage the use of the model in
routine practice, as suggested by the British Thoracic Society (BTS).21
Lee et al.7 also developed a prediction model (Lee model) using simple size metrics
and morphologic features for the differentiation of invasive adenocarcinomas appearing as
SSNs. The model accuracy was reported to be excellent for the identification of invasive
adenocarcinomas. However, it has also not been tested or validated.
Therefore, we aimed to validate the performances of the two prediction models
(Brock and Lee models) for the differentiation of minimally invasive adenocarcinomas
(MIAs) and invasive pulmonary adenocarcinomas (IPAs) from preinvasive lesions among
SSNs. The purpose of our study was to evaluate the feasibility of the two models in the risk
stratification of persistent SSNs.
Page 6 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
METHODS
This retrospective analysis was approved by the institutional review board of Seoul
National University Hospital (IRB No. 1705-116-855) and the requirement for written
informed consent was waived.
Study population
We retrospectively reviewed the electronic medical records of our hospital and found
1915 patients who had undergone surgical resection for lung cancer between 2011 and 2015.
Among the 1915 patients, we identified 1073 patients whose pathologic diagnoses belonged
to the pulmonary adenocarcinoma spectrum including AAH, AIS, MIA, and IPA.1 22
Thereafter, we reviewed the thin-section CT images of the patients to include only those with
SSNs (reviewers: J.S.K., J.H.L., S.Y.A., R.E.Y., H.L., and H.K.); 548 patients whose lung
cancers appeared as solid nodules on CT scans were excluded. We also excluded 76 patients
with nodules smaller than 5 mm or larger than 3 cm and 39 patients in whom data regarding
the family history of lung cancer were not available. Consequently, 410 patients were
included in this study. Among these patients, 18 patients had two nodules and one patient had
three nodules. A single nodule was selected randomly for these 19 patients in order to remove
within-subject correlation. Therefore, a total of 410 nodules from 410 patients were analyzed
in the present study (figure 1). There were 174 men and 236 women (median, 61 years;
interquartile range: 54, 69 years). As for the nodule type, there were 101 pGGNs and 309
PSNs. IPAs were found in 290 nodules followed by MIA in 52 nodules, AIS in 51 nodules,
and AAH in 17 nodules. Median nodule size was 15.8 mm (interquartile range: 11.8, 20.9
mm) (table 1).
Table 1 Demographic data of the entire study population
Page 7 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Characteristics Value
Age (year)a 61 (54, 69)
Sex Male 174 (42.4)
Female 236 (57.6)
Pathology AAH 17 (4.1)
AIS 51 (12.4)
MIA 52 (12.7)
IPA 290 (70.7)
Nodule type Pure GGN 101 (24.6)
Part-solid nodule 309 (75.4)
Family history of lung cancer Yes 16 (3.9)
No 394 (96.1)
Emphysema Yes 44 (10.7)
No 366 (89.3)
Nodule size (mm)a 15.8 (11.8, 20.9)
Solid portion size (mm)a 5.7 (1.6, 11.2)
Solid proportion (%)a 41.0 (11.4, 62.0)
Location Upper lobe 249 (60.7)
Other lobes 161 (39.3)
Lobulation Yes 138 (33.7)
No 272 (66.3)
Spiculation Yes 134 (32.6)
No 276 (67.3)
Nodule count per scana 3 (1, 5)
Note.— Unless otherwise specified, data are numbers of patients (with percentages in
parentheses).
aData are median (with interquartile range in parentheses).
AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass
nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma
Page 8 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Data collection
Patient characteristics including demographic data were collected from the electronic
medical records of Seoul National University Hospital. Patient age, sex, pathologic diagnosis,
family history of lung cancer, and nodule location (lobe) were recorded. The thin-section CT
images were also reviewed to obtain radiologic information of nodules (nodule type, nodule
size, solid portion size, solid proportion, lobulation, spiculation, and nodule count per scan)
and the background lung parenchyma (presence of visually detected emphysema). These
features were used as input variables for logistic regression analysis at the Brock model16
and
Lee model.7 Nodule size and solid portion size were measured as the maximum transverse
diameter (mm) using an electronic caliper. Solid proportion (%) was calculated as the solid
portion size divided by the nodule size. Nodule count was defined as the total number of non-
calcified nodules at least 1 mm in diameter.16
Image review was conducted by three
radiologists (J.P., W.H.L., and H.K.) and each nodule was analyzed once by one of these
radiologists. Details regarding the CT scanning protocols are described in the online
supplementary material.
Measurement variability
We previously analyzed and reported the measurement variability of SSNs and solid
portion size using two same-day repeat CT scans.23
Measurement variability range for the
maximum transverse diameter of SSNs on lung window CT images was ±2.2 mm. For the
solid portion, it was ±3.7 mm. Inter-reader agreement (κ) of nodule type ranged from 0.80 to
0.96. Therefore, we did not re-evaluate the measurement variability or inter-reader agreement
of nodule type in this study.
Statistical analysis
Page 9 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
To investigate whether the variables incorporated in the established models (Brock
and Lee models) were significantly different between preinvasive (AAH and AIS) and
invasive lesions (MIA and IPA), we first performed univariate analysis. Categorical variables
were analyzed using the Pearson Chi-square test or Fisher’s exact test and continuous
variables were analyzed using the independent t-test or Mann-Whitney U test, as appropriate.
We then calculated the predicted probability from each logistic regression model. For
the Brock model, a full model with spiculation was used with input variables of age, sex,
family history of lung cancer, emphysema, nodule size, nodule type, nodule location, nodule
count per scan, and spiculation.16
Regression coefficients and the model constant were
available from the original paper.16
Nodule size was subjected to power transformation prior
to entry as described previously.16
Age and nodule count were centered at a mean of 62 years
and 4, respectively.16
We recorded the predicted probability of each nodule, which was a
continuous value from 0 to 1 (0 to 100%). For the Lee model, two different methods were
used for analysis. For pGGNs, a single cutoff of nodule size (≥10 mm) was used to
discriminate invasive lesions as stated by Lee et al.7 In the case of PSNs, four variables
(nodule size, solid proportion, lobulation, and spiculation) were substituted into the following
regression formula.7
������������� � �
����� � ����� � ��������������� � ���!" � ������#�#������$� % &��!� �
�������������� % ���"" � �����#�'��������
Therefore, predicted probability was obtained only for PSNs in terms of the Lee model. No
preprocessing of variables was performed.
With the predicted probability obtained through each model, receiver operating
Page 10 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
characteristic curve (ROC) analysis was performed to investigate the discriminative
performance of the prediction models in diagnosing invasive lesions. Areas under the ROC
curve (AUCs) were obtained and an optimal cutoff value based on the Youden index was
recorded. We calculated the sensitivity, specificity, accuracy, positive predictive value (PPV),
and negative predictive value (NPV) of each model with the optimal cutoff. For the Brock
model, two different cutoffs were applied to the calculation: 1) a threshold of 10% risk of
malignancy as suggested by the BTS21
and 2) an optimal cutoff based on the Youden index.
ROC analysis was performed for each nodule type separately and then for the entire SSNs in
the case of the Brock model. In terms of the Lee model, ROC analysis was performed only
for PSNs.
AUCs were compared between the models based on DeLong’s method.24
As the
predicted probability of pGGNs was not available for the Lee model, AUC comparison was
conducted only for PSNs. Diagnostic accuracy was also compared between the models using
the McNemar test.
Lastly, calibration of the models was assessed using the Hosmer-Lemeshow test for
the 10 probability groups (deciles). All statistical analyses were performed using two
commercial software programs (MedCalc version 12.3.0, MedCalc Software, Mariakerke,
Belgium; and SPSS 19.0, IBM SPSS Statistics, Armonk, NY, USA) and R software, version
3.1.0 (http://www.R-project.org; PredictABEL package). A P-value <0.05 was considered to
indicate statistical significance.
Page 11 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
RESULTS
Pathologic diagnoses of pGGNs and PSNs
Among 101 pGGNs, 42 were preinvasive and 59 were invasive lesions. As for the
309 PSNs, 26 were preinvasive and 283 were invasive lesions.
Comparisons between preinvasive and invasive lesions
For pGGNs, a family history of lung cancer was more frequently observed in patients
with invasive lesions (6/59) than in those with preinvasive lesions (0/42; P=0.040). Invasive
lesions (14.2±5.4 mm) were also significantly larger than preinvasive lesions (11.1±4.1 mm;
P=0.002). In addition, patients with invasive lesions had a smaller nodule count per scan
(invasive vs. preinvasive lesions: median, 2 vs. 4 nodules per scan; P=0.006). There were no
significant differences in age, sex, presence of emphysema, nodule location, lobulation, and
spiculation (table 2).
Table 2 Comparison of clinical and radiological characteristics between differing pathologic
diagnoses in patients with pure GGNs
Characteristics AAH and AIS
(n=42)
MIA and IPA
(n=59)
P-value
Age (year)a 57±10 57±10 0.839
Sex Male 16 (38.1) 29 (49.2) 0.270
Female 26 (61.9) 30 (50.8)
Family history of lung
cancer
Yes 0 (0) 6 (10.2) 0.040
No 42 (100) 53 (89.8)
Emphysema Yes 5 (11.9) 5 (8.5) 0.738
Page 12 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
No 37 (88.1) 54 (91.5)
Nodule size (mm)a 11.1±4.1 14.2±5.4 0.002
Location Upper lobe 25 (59.5) 31 (52.5) 0.487
Other lobes 17 (40.5) 28 (47.5)
Lobulation Yes 3 (7.1) 12 (20.3) 0.090
No 39 (92.9) 47 (79.7)
Spiculation Yes 1 (2.4) 7 (11.9) 0.135
No 41 (97.6) 52 (88.1)
Nodule count per scanb 4 (2, 6) 2 (1, 3) 0.006
Note.— Unless otherwise specified, data are numbers of patients (with percentages in
parentheses).
aData are mean ± standard deviation.
bData are median (with interquartile range in parentheses).
AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass
nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma
For PSNs, nodule size, solid portion size, and solid proportion were significantly
larger in invasive lesions (invasive vs. preinvasive lesions: median nodule size, 17.6 mm vs.
13.6 mm, P<0.001; median solid portion size, 8.4 mm vs. 4.6 mm, P<0.001; median solid
proportion, 52.8% vs. 36.8%, P=0.032). Lobulation and spiculation were more frequently
observed in invasive lesions (invasive vs. preinvasive lesions: lobulation, 118/283 vs. 5/26,
P=0.025; spiculation, 122/283 vs. 4/26, P=0.006). There were no significant differences in
age, sex, family history of lung cancer, presence of emphysema, nodule location, and nodule
count per scan (table 3).
Table 3 Comparison of clinical and radiological characteristics between differing pathologic
diagnoses in patients with part-solid nodules
Page 13 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Note.
—
Unless
other
wise
specifi
ed,
data
are
numbe
rs of
patient
s
(with
percen
tages
in
parent
heses).
aData are median (with interquartile range in parentheses).
AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass
nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma
SSN risk stratification using the Brock and Lee models
Characteristics AAH and AIS
(n=26)
MIA and IPA
(n=283)
P-value
Age (year)a 62 (54, 68) 63 (56, 70) 0.257
Sex Male 7 (26.9) 122 (43.1) 0.109
Female 19 (73.1) 161 (56.9)
Family history of lung
cancer
Yes 1 (3.8) 9 (3.2) 0.590
No 25 (96.2) 274 (96.8)
Emphysema Yes 2 (7.7) 32 (11.3) 0.752
No 24 (92.3) 251 (88.7)
Nodule size (mm)a 13.6 (9.8,
16.6)
17.6 (13.3,
22.4)
<0.001
Solid portion size
(mm)a
4.6 (3.4, 5.8) 8.4 (5.2, 13.6) <0.001
Solid proportion (%)a 36.8 (25.1,
63.2)
52.8 (33.4,
66.0)
0.032
Location Upper lobe 12 (46.2) 181 (64.0) 0.073
Other lobes 14 (53.8) 102 (36.0)
Lobulation Yes 5 (19.2) 118 (41.7) 0.025
No 21 (80.8) 165 (58.3)
Spiculation Yes 4 (15.4) 122 (43.1) 0.006
No 22 (84.6) 161 (56.9)
Nodule count per scana 3 (2,8) 3 (1,4) 0.132
Page 14 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
For pGGNs, the AUC of the Brock model’s predicted probability for differentiating
invasive lesions from preinvasive lesions was 0.671 [95% confidence interval (CI): 0.571,
0.762] (table 4). A cutoff of 10%, suggested by the BTS, yielded a sensitivity, specificity,
accuracy, PPV, and NPV of 32.2%, 90.5%, 56.4%, 82.6%, and 48.7%, respectively. Another
cutoff of 4.29%, an optimal threshold based on the Youden index, provided a sensitivity,
specificity, accuracy, PPV, and NPV of 64.4%, 64.3%, 64.4%, 71.7%, and 56.3%. Brock
model for pGGNs showed poor calibration (P<0.001). A nodule size cutoff (10 mm)
suggested by Lee et al.7 was also applied to our study population. The resultant sensitivity,
specificity, accuracy, PPV, and NPV were 76.3%, 42.9%, 62.4%, 65.2%, and 56.3%,
respectively. There were no significant differences in diagnostic accuracy between the Brock
model and Lee criteria (Brock model cutoff 10% vs. nodule size cutoff 10mm, P=0.461;
Brock model cutoff 4.29% vs. nodule size cutoff 10 mm, P=0.832).
Table 4 Performance of each prediction model in diagnosing minimally invasive
adenocarcinoma and invasive adenocarcinoma for pure ground-glass nodules
Note.— Data are in percentages except for AUC. There were 42 preinvasive lesions and 59
invasive lesions (minimally invasive adenocarcinomas and invasive adenocarcinomas).
aA cutoff of 10% predicted probability was used as suggested by the British Thoracic Society.
bA cutoff of 4.29% predicted probability, the optimal threshold based on the Youden index,
was used.
Sensitivity Specificity Accuracy PPV NPV AUC
Brock model1a 32.2 90.5 56.4 82.6 48.7 0.671
(0.571, 0.762) Brock model2b 64.4 64.3 64.4 71.7 56.3
Lee model
(Nodule sizec)
76.3 42.9 62.4 65.2 56.3 -
Page 15 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
cA cutoff of 10 mm nodule size was used.
AUC, area under the receiver operating characteristic curve; NPV, negative predictive value;
PPV, positive predictive value
As for PSNs, the AUC of the Brock model was 0.746 (95% CI: 0.694, 0.794) for the
discrimination of invasive lesions from preinvasive lesions (table 5). A cutoff of 10% yielded
a sensitivity, specificity, accuracy, PPV, and NPV of 82.3%, 53.8%, 79.9%, 95.1%, and
21.9%. The optimal cutoff based on the Youden index was 10.11%, which was very close to
the suggested threshold by the BTS. Therefore, performance metrics were not calculated
separately. AUC of the Lee model was 0.771 (95% CI: 0.720, 0.817), and an optimal cutoff of
66.68% provided a sensitivity, specificity, accuracy, PPV, and NPV of 77.0%, 69.2%, 76.4%,
96.5%, and 21.7%, respectively. AUCs and diagnostic accuracies were not significantly
different between the two models (P=0.574 and 0.169, respectively). In addition, both models
exhibited poor calibration (P<0.001).
Table 5 Performance of each prediction model in diagnosing minimally invasive
adenocarcinoma and invasive adenocarcinoma for part-solid nodules
Note.— Data are in percentages except for AUC. There were 26 preinvasive lesions and 283
invasive lesions (minimally invasive adenocarcinomas and invasive adenocarcinomas).
aA cutoff of 10% predicted probability was used, which was suggested by the British
Thoracic Society and was optimal at the same time.
Sensitivity Specificity Accuracy PPV NPV AUC
Brock modela 82.3 53.8 79.9 95.1 21.9 0.746
(0.694, 0.794)
Lee modelb 77.0 69.2 76.4 96.5 21.7 0.771
(0.720, 0.817)
Page 16 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
bAn optimal cutoff of 66.68% was adopted based on the Youden index.
AUC, area under the receiver operating characteristic curve; NPV, negative predictive value;
PPV, positive predictive value
With respect to the pooled analysis for the entire SSNs, the AUC of Brock model was
0.810 (95% CI: 0.769, 0.846). Calibration was also poor in this model (P<0.001).
Page 17 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
DISCUSSION
In this study, we revealed that AUCs for the differentiation of invasive lesions among
PSNs using the established risk prediction models ranged from 0.746 to 0.771 with no
significant differences between the two models. Diagnostic accuracies based on the optimal
cutoffs were 79.9% for the Brock model and 76.4% for the Lee model. For pGGNs, the
diagnostic accuracy was 56.4%-64.4% for the Brock model depending on the cutoff values
used and 62.4% for the Lee criteria. For the entire SSNs, the Brock model showed AUC of
0.810.
McWilliams et al.16
originally developed a lung cancer prediction model (Brock
model) using participants enrolled in a lung cancer screening study. Thus, the Brock model
initially targeted pulmonary nodules detected on first screening CT. Incidentally detected
nodules as well as surgical candidates were not the original target lesions of this model.
However, at present, the BTS recommends using the same diagnostic approach for nodules
detected incidentally as those detected through screening.21
BTS also recommends using the
Brock model for the risk calculation of both solid and subsolid nodules.21
A cutoff of 10%
predicted probability for malignancy is suggested in order to differentiate high risk SSNs for
the performance of biopsy or surgical resection.21
This quantitative diagnostic approach is to
discern malignant SSNs with an appropriate false positive rate. However, it must be noted
that most persistent SSNs belong to one of the four categories of the adenocarcinoma
spectrum: AAH, AIS, MIA and IPA. Therefore, the potential of the risk prediction model in
discriminating lesions with invasive components (MIA and IPA) should also be tested using
pathologic diagnosis as a reference standard. Indeed, as clinical management strategies differ
substantially between preinvasive and invasive lesions, if it can be feasible to predict the
Page 18 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
invasiveness of SSNs, clinical planning whether to perform annual CT surveillance, limited
resection, or conventional lobectomy can be facilitated.
The performance of the prediction models for the risk stratification of SSNs was not
optimal according to our study results. For PSNs, AUCs of the two models ranged between
0.746 and 0.771 with diagnostic accuracies close to 80%. The performance of the prediction
model for PSNs in the original paper by Lee et al.7 was 0.905 (AUC). The study population
of the present study was similar to that of the study by Lee et al.7 An important reason for the
performance drop would be the spectrum effect, which is a common cause of model
performance heterogeneity.25
A variation in the assessment of CT morphologic features
(lobulation and spiculation) would be another potential cause. Past research on distinguishing
invasive adenocarcinomas appearing as SSNs have reported that logistic regression models
built with size metrics, morphologic features, or texture features showed AUCs ranging from
0.79 to 0.98.4-6 9 11 13
However, these models were not tested for an independent cohort or
validated externally.
Another important finding of our study was that the PPV for the differentiation of
invasive lesions among PSNs was very high for both models, over 95%. In other words, the
probability of being an invasive lesion was over 95% for nodules predicted as being invasive
through these models. A concern, however, is the high false negative rate of these models.
PSNs predicted as preinvasive lesions, which have a low calculated risk, should be managed
according to their solid portion size, if they are persistent lesions.26
A few studies have shown
that the solid portions in PSNs are well correlated to the pathologic invasive component.12 27
28 Fleischner Society guideline recommends that PSNs with solid components ≥6 mm be
monitored with CT scans at 3-6 months interval.26
PSNs with solid portions larger than 8 mm
should be biopsied or surgically resected in consideration of invasive adenocarcinomas.26
Page 19 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
BTS also recommends that the solid component size be considered to further refine the
estimate of malignancy risk.21
In addition, growing solid component is also a sign of an
invasive adenocarcinoma as described in both guidelines.21 26
The diagnostic accuracies of both the Brock model and Lee criteria were even lower
for pGGNs. Among multiple clinical and radiologic characteristics investigated in our study,
only three variables (family history of lung cancer, nodule size, and nodule count per scan)
were significantly different between preinvasive and invasive lesions. This implies the need
for other useful features for the development of new better prediction models. Features such
as nodule volume, mass, or radiomic features may provide additional clues for their
differentiation.4 29
In addition, changes in nodule characteristics at follow-up CT scans, such
as an increase in nodule size, attenuation, or new development of a solid portion, may also be
valuable for the discrimination.14
Alternatively, computational classification analysis
including deep learning algorithms, which do not require hand-crafted features and can be
self-trained directly from raw image pixels, may be another solution for the diagnosis of
pGGNs.30
The Brock model has been externally validated for the cohorts of the Danish Lung
Cancer Screening Trial (DLCST)19
and National Lung Screening Trial (NLST);18
AUCs for
the discrimination of malignant from benign nodules ranged from 0.834 for the former and
0.963 for the latter. AUCs for the validation cohort of the original paper, the British Columbia
Cancer Agency (BCCA) chemoprevention trial cohort, was 0.970.16
In addition, for an
Australian lung cancer screening cohort, Zhao et al.20
tested the utility of the Brock model for
the baseline evaluation of 52 SSNs and demonstrated that the AUC was 0.89. To the contrary,
however, the model performance evaluated in our study was lower than those reported in the
literature. The main reason for such a discrepancy may be that we included patients who
Page 20 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
underwent surgical resection of SSNs unlike previous studies. Thus, the proportion of
preinvasive lesions was small (16.6%) and a major portion of our study population consisted
of invasive lesions (83.4%). Such high prevalence of invasive lesions would have affected
our study results. Nevertheless, SSNs of interest in daily clinical practice may be closer to
those in our study. In routine practice, transient SSNs, which are definitely benign, do not
require risk calculation as they are easily confirmed through follow-up CT scans at short-term
intervals.26
In addition, small SSNs <6 mm are usually preinvasive and do not require CT
surveillance. On the other hand, particular concern should be given to persistent SSNs ≥6 mm,
especially to those with solid components. As the role of biopsy or positron emission
tomography is limited for SSNs,15
we supposed that risk prediction models may provide
value for more appropriate management planning. In this context, we applied prediction
models to surgically resected SSNs for the validation of their clinical utility.
There were several limitations to our study. First, our study was not conducted for a
screening cohort as described earlier in this manuscript. The prevalence of preinvasive lesions
was low compared to that of the screening setting. Thus, the performance measures in this
study should be carefully interpreted with respect to the target population, which were the
incidentally detected surgical candidates. Second, our retrospective study included a small
number of patients and analyses were conducted separately for pGGNs and PSNs. Separate
analysis of pGGNs and PSNs has resulted in a slight underestimation of the performance of
Brock model. Third, optimal cutoffs for the models were not obtained from ROC analyses of
the original study populations from which the models were derived. Fourth, radiologic nodule
information was extracted from our heterogeneous CT dataset, in which CT acquisition
parameters such as radiation dosage, slice thickness or contrast-enhancement were not
uniform across the study population. However, these factors would have had little effect on
Page 21 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
the variables we used. In addition, all CT scans had thin-section images (slice thickness ≤1.5
mm). Fifth, nodule size as well as solid portion size were measured as the longest transverse
diameter in accordance with the definition of lesion size and solid proportion in the original
papers. However, recent analyses have revealed that the usage of average diameter as an
input variable may enhance the model performance.31
In conclusion, the performance of the Brock model and Lee model for the
differentiation of invasive lesions among SSNs was suboptimal. In particular, both models
showed lower performance for pGGNs compared to that for PSNs. Thus, an alternative
approach such as computer-aided classification should be developed for the pre-operative
diagnosis of invasive lesions among SSNs.
Page 22 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Acknowledgements
We would like to thank Chris Woo, B.A., for editorial assistance.
Contributors
HK and CMP contributed to conception and design; HK, SJ, JHL, SYA, REY, HL, JP,
WHL, EJH, SML and JMG contributed to acquisition of data, or analysis and interpretation
of data; HK, CMP and JMG were involved in drafting the manuscript or revising it critically
for important intellectual content; all authors gave approval of the final version of the
manuscript.
Funding
This study was supported by Basic Science Research Program through the National
Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT & Future
Planning (grant number: 2017R1A2B4008517). Funder had no role in the study design; in the
collection, analysis and interpretation of the data; in the writing of the report; and in the
decision to submit the paper for publication.
Competing interests
None declared.
Patient consent
The requirement for written informed consent was waived by the Institutional
Review Board.
Page 23 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Ethics approval
Institutional Review Board of Seoul National University Hospital.
Data sharing statement
All data are available from the corresponding author.
Page 24 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
REFERENCES
1. Austin JH, Garg K, Aberle D, et al. Radiologic implications of the 2011 classification of
adenocarcinoma of the lung. Radiology 2013;266:62-71.
2. Yankelevitz DF, Yip R, Smith JP, et al. CT Screening for lung cancer: nonsolid nodules in
baseline and annual repeat rounds. Radiology 2015;277:555-64.
3. Scholten ET, de Jong PA, de Hoop B, et al. Towards a close computed tomography
monitoring approach for screen detected subsolid pulmonary nodules? Eur Respir J
2015;45:765-73.
4. Chae HD, Park CM, Park SJ, et al. Computerized texture analysis of persistent part-solid
ground-glass nodules: differentiation of preinvasive lesions from invasive pulmonary
adenocarcinomas. Radiology 2014;273:285-93.
5. Ding H, Shi J, Zhou X, et al. Value of CT characteristics in predicting invasiveness of
adenocarcinoma presented as pulmonary ground-glass nodules. Thorac Cardiovasc
Surg 2017;65:136-41.
6. Jin C, Cao J, Cai Y, et al. A nomogram for predicting the risk of invasive pulmonary
adenocarcinoma for patients with solitary peripheral subsolid nodules. J Thorac
Cardiovasc Surg 2017;153:462-69 e1.
7. Lee SM, Park CM, Goo JM, et al. Invasive pulmonary adenocarcinomas versus preinvasive
lesions appearing as ground-glass nodules: differentiation by using CT features.
Radiology 2013;268:265-73.
8. Li Q, Fan L, Cao ET, et al. Quantitative CT analysis of pulmonary pure ground-glass
nodule predicts histological invasiveness. Eur J Radiol 2017;89:67-71.
9. Liang J, Xu XQ, Xu H, et al. Using the CT features to differentiate invasive pulmonary
adenocarcinoma from pre-invasive lesion appearing as pure or mixed ground-glass
Page 25 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
nodules. Br J Radiol 2015;88:20140811.
10. Moon Y, Sung SW, Lee KY, et al. Pure ground-glass opacity on chest computed
tomography: predictive factors for invasive adenocarcinoma. J Thorac Dis
2016;8:1561-70.
11. Son JY, Lee HY, Kim JH, et al. Quantitative CT analysis of pulmonary ground-glass
opacity nodules for distinguishing invasive adenocarcinoma from non-invasive or
minimally invasive adenocarcinoma: the added value of using iodine mapping. Eur
Radiol 2016;26:43-54.
12. Yanagawa M, Johkoh T, Noguchi M, et al. Radiological prediction of tumor invasiveness
of lung adenocarcinoma on thin-section CT. Medicine (Baltimore) 2017;96:e6331.
13. Zhang Y, Shen Y, Qiang JW, et al. HRCT features distinguishing pre-invasive from
invasive pulmonary adenocarcinomas appearing as ground-glass nodules. Eur Radiol
2016;26:2921-8.
14. Kim H, Park CM, Koh JM, et al. Pulmonary subsolid nodules: what radiologists need to
know about the imaging features and management strategy. Diagn Interv Radiol
2014;20:47-57.
15. Naidich DP, Bankier AA, MacMahon H, et al. Recommendations for the management of
subsolid pulmonary nodules detected at CT: a statement from the Fleischner Society.
Radiology 2013;266:304-17.
16. McWilliams A, Tammemagi MC, Mayo JR, et al. Probability of cancer in pulmonary
nodules detected on first screening CT. N Engl J Med 2013;369:910-9.
17. Al-Ameri A, Malhotra P, Thygesen H, et al. Risk of malignancy in pulmonary nodules: a
validation study of four prediction models. Lung Cancer 2015;89:27-30.
18. White CS, Dharaiya E, Campbell E, et al. The Vancouver lung cancer risk prediction
model: assessment by using a subset of the National Lung Screening Trial cohort.
Page 26 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Radiology 2017;283:264-72.
19. Winkler Wille MM, van Riel SJ, Saghir Z, et al. Predictive accuracy of the PanCan lung
cancer risk prediction model -external validation based on CT from the Danish Lung
Cancer Screening Trial. Eur Radiol 2015;25:3093-9.
20. Zhao H, Marshall HM, Yang IA, et al. Screen-detected subsolid pulmonary nodules: long-
term follow-up and application of the PanCan lung cancer risk prediction model. Br J
Radiol 2016;89:20160016.
21. Callister ME, Baldwin DR, Akram AR, et al. British Thoracic Society guidelines for the
investigation and management of pulmonary nodules. Thorax 2015;70 Suppl 2:ii1-
ii54.
22. Travis WD, Brambilla E, Noguchi M, et al. International association for the study of lung
cancer/American thoracic society/European respiratory society international
multidisciplinary classification of lung adenocarcinoma. J Thorac Oncol 2011;6:244-
85.
23. Kim H, Park CM, Song YS, et al. Measurement variability of persistent pulmonary
subsolid nodules on same-day repeat CT: what is the threshold to determine true
nodule growth during follow-up? PLoS One 2016;11:e0148853.
24. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more
correlated receiver operating characteristic curves: a nonparametric approach.
Biometrics 1988;44:837-45.
25. Riley RD, Ensor J, Snell KI, et al. External validation of clinical prediction models using
big datasets from e-health records or IPD meta-analysis: opportunities and challenges.
BMJ 2016;353:i3140.
26. MacMahon H, Naidich DP, Goo JM, et al. Guidelines for management of incidental
pulmonary nodules detected on CT images: from the Fleischner Society 2017.
Page 27 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Radiology 2017;284:228-43.
27. Ko JP, Suh J, Ibidapo O, et al. Lung adenocarcinoma: correlation of quantitative CT
findings with pathologic findings. Radiology 2016;280:931-9.
28. Lee KH, Goo JM, Park SJ, et al. Correlation between the size of the solid component on
thin-section CT and the invasive component on pathology in small lung
adenocarcinomas manifesting as ground-glass nodules. J Thorac Oncol 2014;9:74-82.
29. Song YS, Park CM, Park SJ, et al. Volume and mass doubling times of persistent
pulmonary subsolid nodules detected in patients without known malignancy.
Radiology 2014;273:276-84.
30. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer
with deep neural networks. Nature 2017;542:115-18.
31. van Riel SJ, Ciompi F, Jacobs C, et al. Malignancy risk estimation of screen-detected
nodules at baseline CT: comparison of the PanCan model, Lung-RADS and NCCN
guidelines. Eur Radiol 2017;27:4019-29.
Page 28 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
FIGURE LEGENDS
Figure 1 Flow chart of patient inclusion and exclusion.
pGGN, pure ground-glass nodule; PSN, part-solid nodule; SSN, subsolid nodule
Page 29 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Flow chart of patient inclusion and exclusion.
127x157mm (600 x 600 DPI)
Page 30 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
CT SCANNING PROTOCOLS FOR THE STUDY POPULATION
Patients underwent CT scans using seven different scanners from four vendors. CT scans in our study were performed with the
following scanning protocols.
Vendor Toshibaa Siemens
b Siemens
b GE
c GE
c Philips
d Philips
d
Machine Aquilion One Definition Sensation
16
LightSpeed
Ultra
Discovery
CT750 HD
Brilliance 64 Ingenuity
kVp 120 120 120 120 120 120 120
mAs Target standard
deviation of
noise, 14
Quality
reference
mAs, 150
Quality
reference
mAs, 100
Noise index,
16.36
Noise
index, 22.67
Reference
mAs, 200
Reference
mAs, 170
Automatic
exposure
control
Sure Exposure CARE
Dose 4D
CARE
Dose 4D
Auto mA Auto mA Doseright
ACS, Z
DOM
Doseright
ACS, Z
DOM
Rotation time
(sec)
0.5 0.5 0.5 0.5 0.5 0.5 0.5
Collimation
(mm)
0.5x100 0.6x64 0.75x16 1.25x8 0.625x64 0.625x64 0.625x64
Slice thickness
(mm)
1 1 1 1 1.25 1 1
Pitch HP 103 1 1 0.875 0.984 1.014 1.0015
Page 31 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Kernel FC 15-H B60f B60f Bone Chest YC 0 YC 0
Increment 1 1 1 1 1.25 1 1
Matrix 512x512 512x512 512x512 512x512 512x512 512x512 512x512
* All patients were scanned in supine position from the lung apex to the base during suspended maximum inspiration.
* For contrast enhancement, a total of 70-90 mL of 370 mgI/mL of the contrast material, iopamidol (Pamiray 370; Dongkook Pharmaceutical,
Seoul, Korea) or iopromide (Ultravist 370; Schering, Berlin, Germany), was injected at a rate of 2.0-2.3mL/sec using a power injector. CT
scanning was performed with a 60-second delay.
aToshiba Medical Systems, Otawara, Japan
bSiemens Healthcare, Forchheim, Germany
cGE Healthcare, Waukesha, WI, USA
dPhilips Healthcare, Cleveland, OH, USA
Page 32 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
TRIPOD Checklist: Prediction Model Validation
Section/Topic Item Checklist Item Page
Title and abstract
Title 1 Identify the study as developing and/or validating a multivariable prediction model, the target population, and the outcome to be predicted.
1
Abstract 2 Provide a summary of objectives, study design, setting, participants, sample size, predictors, outcome, statistical analysis, results, and conclusions.
3
Introduction
Background and objectives
3a Explain the medical context (including whether diagnostic or prognostic) and rationale for developing or validating the multivariable prediction model, including references to existing models.
5 (line 17), 6 (line 3)
3b Specify the objectives, including whether the study describes the development or validation of the model or both.
6 (line 7)
Methods
Source of data
4a Describe the study design or source of data (e.g., randomized trial, cohort, or registry data), separately for the development and validation data sets, if applicable.
7 (line 6-7)
4b Specify the key study dates, including start of accrual; end of accrual; and, if applicable, end of follow-up.
7 (line 7)
Participants
5a Specify key elements of the study setting (e.g., primary care, secondary care, general population) including number and location of centres.
2 (line 8), 9 (line 3)
5b Describe eligibility criteria for participants. 7
5c Give details of treatments received, if relevant. N/A
Outcome 6a
Clearly define the outcome that is predicted by the prediction model, including how and when assessed.
10 (line 2-3)
6b Report any actions to blind assessment of the outcome to be predicted. -
Predictors
7a Clearly define all predictors used in developing or validating the multivariable prediction model, including how and when they were measured.
9
7b Report any actions to blind assessment of predictors for the outcome and other predictors.
-
Sample size 8 Explain how the study size was arrived at. -
Missing data 9 Describe how missing data were handled (e.g., complete-case analysis, single imputation, multiple imputation) with details of any imputation method.
-
Statistical analysis methods
10c For validation, describe how the predictions were calculated. 10
10d Specify all measures used to assess model performance and, if relevant, to compare multiple models.
10-11
10e Describe any model updating (e.g., recalibration) arising from the validation, if done. N/A
Risk groups 11 Provide details on how risk groups were created, if done. N/A
Development vs. validation
12 For validation, identify any differences from the development data in setting, eligibility criteria, outcome, and predictors.
18, 21
Results
Participants
13a Describe the flow of participants through the study, including the number of participants with and without the outcome and, if applicable, a summary of the follow-up time. A diagram may be helpful.
12
13b Describe the characteristics of the participants (basic demographics, clinical features, available predictors), including the number of participants with missing data for predictors and outcome.
8
13c For validation, show a comparison with the development data of the distribution of important variables (demographics, predictors and outcome).
-
Model performance
16 Report performance measures (with CIs) for the prediction model. 15-17
Model-updating 17 If done, report the results from any model updating (i.e., model specification, model performance).
N/A
Discussion
Limitations 18 Discuss any limitations of the study (such as nonrepresentative sample, few events per predictor, missing data).
21
Interpretation
19a For validation, discuss the results with reference to performance in the development data, and any other validation data.
19-20
19b Give an overall interpretation of the results, considering objectives, limitations, results from similar studies, and other relevant evidence.
22
Implications 20 Discuss the potential clinical use of the model and implications for future research. 19
Other information
Page 33 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
TRIPOD Checklist: Prediction Model Validation
Supplementary information 21
Provide information about the availability of supplementary resources, such as study protocol, Web calculator, and data sets.
24 (line 2)
Funding 22 Give the source of funding and the role of the funders for the present study.
23 (line 9-11)
We recommend using the TRIPOD Checklist in conjunction with the TRIPOD Explanation and Elaboration document.
Page 34 of 34
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Validation of prediction models for risk stratification of
incidentally detected pulmonary subsolid nodules: a
retrospective cohort study in a Korean tertiary medical
center
Journal: BMJ Open
Manuscript ID bmjopen-2017-019996.R2
Article Type: Research
Date Submitted by the Author: 28-Mar-2018
Complete List of Authors: Kim, Hyungjin; Seoul National University Hospital Park, Chang Min; Department of Radiology Jeon, Sun Kyung; Seoul National University Hospital Lee, Jong Hyuk; Seoul National University Hospital Ahn, Su Yeon; Seoul National University Hospital Yoo, Roh-Eul; Seoul National University Hospital Lim, Hyun-ju; National Cancer center Park, Juil; Seoul National University Hospital Lim, Woo Hyeon; Seoul National University Hospital Hwang, Eui Jin; Seoul National University Hospital
Lee, Sang Min; University of Ulsan College of Medicine Goo, Jin Mo; Seoul National University Hospital
<b>Primary Subject Heading</b>:
Oncology
Secondary Subject Heading: Diagnostics
Keywords: subsolid nodule, prediction model, logistic model, external validation, adenocarcinoma, Brock model
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open on F
ebruary 5, 2020 by guest. Protected by copyright.
http://bmjopen.bm
j.com/
BM
J Open: first published as 10.1136/bm
jopen-2017-019996 on 24 May 2018. D
ownloaded from
For peer review only
Validation of prediction models for risk stratification of
incidentally detected pulmonary subsolid nodules: a
retrospective cohort study in a Korean tertiary medical
center
Hyungjin Kim1, Chang Min Park
1, 2, 3, Sunkyung Jeon
1, Jong Hyuk Lee
1, Su Yeon Ahn
1, Roh-
Eul Yoo1, Hyun-ju Lim
1, 4, Juil Park
1, Woo Hyeon Lim
1, Eui Jin Hwang
1, Sang Min Lee
5, Jin
Mo Goo1, 2, 3
1Department of Radiology, Seoul National University College of Medicine, Seoul, Korea
2Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul,
Korea
3Seoul National University Cancer Research Institute, Seoul, Korea
4Department of Radiology, National Cancer center, Goyang, Korea
5Department of Radiology and Research Institute of Radiology, University of Ulsan College
of Medicine, Seoul, Korea
* Corresponding author
Chang Min Park, MD, PhD. Department of Radiology, Seoul National University College
of Medicine, 101, Daehak-ro, Jongno-gu, Seoul 110-744, Korea. Tel: 82-2-2072-0367, Fax:
Page 1 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
82-2-743-7418. E-mail: [email protected]
Word count: 3339
Page 2 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
ABSTRACT
Objectives: To validate the performances of two prediction models (Brock and Lee models)
for the differentiation of minimally-invasive and invasive pulmonary adenocarcinomas
(MIA/IPA) from preinvasive lesions among subsolid nodules (SSNs).
Design: A retrospective cohort study.
Setting: A tertiary university hospital in South Korea.
Participants: 410 patients with 410 incidentally detected SSNs who underwent surgical
resection for the pulmonary adenocarcinoma spectrum between 2011 and 2015.
Primary and secondary outcome measures: Using clinical and radiological variables, the
predicted probability of MIA/IPA was calculated from pre-existing logistic models (Brock
and Lee models). Areas under the receiver operating characteristic curve (AUCs) were
calculated and compared between models. Performance metrics including sensitivity,
specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV)
were also obtained.
Results: For pure ground-glass nodules (n=101), the AUC of the Brock model in
differentiating MIA/IPA (59/101) from preinvasive lesions (42/101) was 0.671. Sensitivity,
specificity, accuracy, PPV, and NPV based on the optimal cutoff value were 64.4%, 64.3%,
64.4%, 71.7%, and 56.3%, respectively. Sensitivity, specificity, accuracy, PPV, and NPV
according to the Lee criteria were 76.3%, 42.9%, 62.4%, 65.2%, and 56.3%, respectively.
AUC was not obtained for the Lee model as a single cutoff of nodule size (≥10 mm) was
suggested by this model for the assessment of pure ground-glass nodules. For part-solid
nodules (n=309; 26 preinvasive lesions and 283 MIA /IPAs), the AUC was 0.746 for the
Page 3 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Brock model and 0.771 for the Lee model (P=0.574). Sensitivity, specificity, accuracy, PPV,
and NPV were 82.3%, 53.8%, 79.9%, 95.1%, and 21.9%, respectively, for the Brock model
and 77.0%, 69.2%, 76.4%, 96.5%, and 21.7%, respectively, for the Lee model.
Conclusions: The performance of prediction models for the incidentally detected SSNs in
differentiating MIA/IPA from preinvasive lesions might be suboptimal. Thus, an alternative
risk calculation model is required for the incidentally detected SSNs.
Strengths and limitations of this study
• This is the first study to externally validate the performance of pre-existing risk prediction
models for the incidentally detected pulmonary subsolid nodules.
• This study performed head-to-head comparisons between the prediction models for the risk
stratification of subsolid nodules.
• The main limitation of this study is that it only analyzed surgically resected lung nodules,
thus inducing selection bias.
• Study population was small to conduct separate analyses for the pure ground-glass nodules
and part-solid nodules.
Page 4 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
INTRODUCTION
Pulmonary subsolid nodules (SSNs) represent a histologic spectrum of
adenocarcinoma and its preinvasive precursors, including atypical adenomatous hyperplasia
(AAH) and adenocarcinoma-in-situ (AIS).1 SSNs are common findings at chest computed
tomography (CT) which have been increasingly detected in CT screening studies.2 3
Indeed,
according to one prospective screening study, 4.2% of the participants had at least one pure
ground-glass nodule (pGGN) and 5.0% had at least one part-solid nodule (PSN) at baseline
rounds of screening.2
With this prevalence in mind, numerous studies have justifiably focused on the
differentiation of invasive adenocarcinomas from preinvasive lesions4-13
as invasive
adenocarcinoma requires surgical resection with conventional lobectomy and lymph node
dissection14
whereas preinvasive lesions can be followed-up conservatively with annual CT
surveillance or resected at a lesser extent (sublobar resection).15
Thus, the discrimination of
invasive adenocarcinoma has been a major topic of interest for many radiologists and
clinicians to date.
In a quest to obtain quantitative risk prediction tools for pulmonary nodules,
McWilliams et al.16
developed a prediction model (Brock model) utilizing various clinical
and radiological features. The Brock model demonstrated higher accuracy in determining the
likelihood of malignancy in pulmonary nodules compared to other existing models17
and was
also externally validated in three independent screening populations.18-20
Nevertheless, in the
context that a substantial percentage of persistent SSNs may belong to the adenocarcinoma
spectrum, the performance of the established model in differentiating invasive
Page 5 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
adenocarcinoma should also be validated in order to encourage the use of the model in
routine practice, as suggested by the British Thoracic Society (BTS).21
Lee et al.7 also developed a prediction model (Lee model) using simple size metrics
and morphologic features for the differentiation of invasive adenocarcinomas appearing as
SSNs. The model accuracy was reported to be excellent for the identification of invasive
adenocarcinomas. However, it has also not been tested or validated.
Therefore, we aimed to validate the performances of the two prediction models
(Brock and Lee models) for the differentiation of minimally invasive adenocarcinomas
(MIAs) and invasive pulmonary adenocarcinomas (IPAs) from preinvasive lesions among
SSNs. The purpose of our study was to evaluate the feasibility of the two models in the risk
stratification of persistent SSNs.
Page 6 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
METHODS
This retrospective analysis was approved by the institutional review board of Seoul
National University Hospital (IRB No. 1705-116-855) and the requirement for written
informed consent was waived.
Study population
We retrospectively reviewed the electronic medical records of our hospital and found
1915 patients who had undergone surgical resection for lung cancer between 2011 and 2015.
Among the 1915 patients, we identified 1073 patients whose pathologic diagnoses belonged
to the pulmonary adenocarcinoma spectrum including AAH, AIS, MIA, and IPA.1 22
Thereafter, we reviewed the thin-section CT images of the patients to include only those with
SSNs (reviewers: J.S.K., J.H.L., S.Y.A., R.E.Y., H.L., and H.K.); 548 patients whose lung
cancers appeared as solid nodules on CT scans were excluded. We also excluded 76 patients
with nodules smaller than 5 mm or larger than 3 cm and 39 patients in whom data regarding
the family history of lung cancer were not available. Consequently, 410 patients were
included in this study. Among these patients, 18 patients had two nodules and one patient had
three nodules. A single nodule was selected randomly for these 19 patients in order to remove
within-subject correlation. Therefore, a total of 410 nodules from 410 patients were analyzed
in the present study (figure 1). There were 174 men and 236 women (median, 61 years;
interquartile range: 54, 69 years). As for the nodule type, there were 101 pGGNs and 309
PSNs. IPAs were found in 290 nodules followed by MIA in 52 nodules, AIS in 51 nodules,
and AAH in 17 nodules. Median nodule size was 15.8 mm (interquartile range: 11.8, 20.9
mm) (table 1).
Table 1 Demographic data of the entire study population
Page 7 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Characteristics Value
Age (year)a 61 (54, 69)
Sex Male 174 (42.4)
Female 236 (57.6)
Pathology AAH 17 (4.1)
AIS 51 (12.4)
MIA 52 (12.7)
IPA 290 (70.7)
Nodule type Pure GGN 101 (24.6)
Part-solid nodule 309 (75.4)
Family history of lung cancer Yes 16 (3.9)
No 394 (96.1)
Emphysema Yes 44 (10.7)
No 366 (89.3)
Nodule size (mm)a 15.8 (11.8, 20.9)
Solid portion size (mm)a 5.7 (1.6, 11.2)
Solid proportion (%)a 41.0 (11.4, 62.0)
Location Upper lobe 249 (60.7)
Other lobes 161 (39.3)
Lobulation Yes 138 (33.7)
No 272 (66.3)
Spiculation Yes 134 (32.6)
No 276 (67.3)
Nodule count per scana 3 (1, 5)
Note.— Unless otherwise specified, data are numbers of patients (with percentages in
parentheses).
aData are median (with interquartile range in parentheses).
AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass
nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma
Page 8 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Data collection
Patient characteristics including demographic data were collected from the electronic
medical records of Seoul National University Hospital. Patient age, sex, pathologic diagnosis,
family history of lung cancer, and nodule location (lobe) were recorded. The thin-section CT
images were also reviewed to obtain radiologic information of nodules (nodule type, nodule
size, solid portion size, solid proportion, lobulation, spiculation, and nodule count per scan)
and the background lung parenchyma (presence of visually detected emphysema). These
features were used as input variables for logistic regression analysis at the Brock model16
and
Lee model.7 Nodule size and solid portion size were measured as the maximum transverse
diameter (mm) using an electronic caliper. Solid proportion (%) was calculated as the solid
portion size divided by the nodule size. Nodule count was defined as the total number of non-
calcified nodules at least 1 mm in diameter.16
Image review was conducted by three
radiologists (J.P., W.H.L., and H.K.) and each nodule was analyzed once by one of these
radiologists. Details regarding the CT scanning protocols are described in the online
supplementary material.
Measurement variability
We previously analyzed and reported the measurement variability of SSNs and solid
portion size using two same-day repeat CT scans.23
Measurement variability range for the
maximum transverse diameter of SSNs on lung window CT images was ±2.2 mm. For the
solid portion, it was ±3.7 mm. Inter-reader agreement (κ) of nodule type ranged from 0.80 to
0.96. Therefore, we did not re-evaluate the measurement variability or inter-reader agreement
of nodule type in this study.
Statistical analysis
Page 9 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
To investigate whether the variables incorporated in the established models (Brock
and Lee models) were significantly different between preinvasive (AAH and AIS) and
invasive lesions (MIA and IPA), we first performed univariate analysis. Categorical variables
were analyzed using the Pearson Chi-square test or Fisher’s exact test and continuous
variables were analyzed using the independent t-test or Mann-Whitney U test, as appropriate.
We then calculated the predicted probability from each logistic regression model. For
the Brock model, a full model with spiculation was used with input variables of age, sex,
family history of lung cancer, emphysema, nodule size, nodule type, nodule location, nodule
count per scan, and spiculation.16
Regression coefficients and the model constant were
available from the original paper.16
Nodule size was subjected to power transformation prior
to entry as described previously.16
Age and nodule count were centered at a mean of 62 years
and 4, respectively.16
We recorded the predicted probability of each nodule, which was a
continuous value from 0 to 1 (0 to 100%). For the Lee model, two different methods were
used for analysis. For pGGNs, a single cutoff of nodule size (≥10 mm) was used to
discriminate invasive lesions as stated by Lee et al.7 In the case of PSNs, four variables
(nodule size, solid proportion, lobulation, and spiculation) were substituted into the following
regression formula.7
Logit�Probability� =
0.396 − 0.200 × �nodulesize�mm� − 0.048 × �solidproportion�%� + 1.049 ×
�nonlobulation� + 3.288 × �nonspiculation�
Therefore, predicted probability was obtained only for PSNs in terms of the Lee model. No
preprocessing of variables was performed.
With the predicted probability obtained through each model, receiver operating
Page 10 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
characteristic curve (ROC) analysis was performed to investigate the discriminative
performance of the prediction models in diagnosing invasive lesions. Areas under the ROC
curve (AUCs) were obtained and an optimal cutoff value based on the Youden index was
recorded. We calculated the sensitivity, specificity, accuracy, positive predictive value (PPV),
and negative predictive value (NPV) of each model with the optimal cutoff. For the Brock
model, two different cutoffs were applied to the calculation: 1) a threshold of 10% risk of
malignancy as suggested by the BTS21
and 2) an optimal cutoff based on the Youden index.
ROC analysis was performed for each nodule type separately and then for the entire SSNs in
the case of the Brock model. In terms of the Lee model, ROC analysis was performed only
for PSNs.
AUCs were compared between the models based on DeLong’s method.24
As the
predicted probability of pGGNs was not available for the Lee model, AUC comparison was
conducted only for PSNs. Diagnostic accuracy was also compared between the models using
the McNemar test.
Lastly, calibration of the models was assessed using the Hosmer-Lemeshow test for
the 10 probability groups (deciles). All statistical analyses were performed using two
commercial software programs (MedCalc version 12.3.0, MedCalc Software, Mariakerke,
Belgium; and SPSS 19.0, IBM SPSS Statistics, Armonk, NY, USA) and R software, version
3.1.0 (http://www.R-project.org; PredictABEL package). A P-value <0.05 was considered to
indicate statistical significance.
Patient and public involvement
Patients or public were not involved in the development of the research question and
outcome measures. No patients were involved in the study design or conduct of the study.
Page 11 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Dissemination of the study results to the study participants is not practical given the
retrospective nature of our study. Lastly, there were no patient advisers.
Page 12 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
RESULTS
Pathologic diagnoses of pGGNs and PSNs
Among 101 pGGNs, 42 were preinvasive and 59 were invasive lesions. As for the
309 PSNs, 26 were preinvasive and 283 were invasive lesions.
Comparisons between preinvasive and invasive lesions
For pGGNs, a family history of lung cancer was more frequently observed in patients
with invasive lesions (6/59) than in those with preinvasive lesions (0/42; P=0.040). Invasive
lesions (14.2±5.4 mm) were also significantly larger than preinvasive lesions (11.1±4.1 mm;
P=0.002). In addition, patients with invasive lesions had a smaller nodule count per scan
(invasive vs. preinvasive lesions: median, 2 vs. 4 nodules per scan; P=0.006). There were no
significant differences in age, sex, presence of emphysema, nodule location, lobulation, and
spiculation (table 2).
Table 2 Comparison of clinical and radiological characteristics between differing pathologic
diagnoses in patients with pure GGNs
Characteristics AAH and AIS
(n=42)
MIA and IPA
(n=59)
P-value
Age (year)a 57±10 57±10 0.839
Sex Male 16 (38.1) 29 (49.2) 0.270
Female 26 (61.9) 30 (50.8)
Family history of lung
cancer
Yes 0 (0) 6 (10.2) 0.040
No 42 (100) 53 (89.8)
Emphysema Yes 5 (11.9) 5 (8.5) 0.738
Page 13 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
No 37 (88.1) 54 (91.5)
Nodule size (mm)a 11.1±4.1 14.2±5.4 0.002
Location Upper lobe 25 (59.5) 31 (52.5) 0.487
Other lobes 17 (40.5) 28 (47.5)
Lobulation Yes 3 (7.1) 12 (20.3) 0.090
No 39 (92.9) 47 (79.7)
Spiculation Yes 1 (2.4) 7 (11.9) 0.135
No 41 (97.6) 52 (88.1)
Nodule count per scanb 4 (2, 6) 2 (1, 3) 0.006
Note.— Unless otherwise specified, data are numbers of patients (with percentages in
parentheses).
aData are mean ± standard deviation.
bData are median (with interquartile range in parentheses).
AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass
nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma
For PSNs, nodule size, solid portion size, and solid proportion were significantly
larger in invasive lesions (invasive vs. preinvasive lesions: median nodule size, 17.6 mm vs.
13.6 mm, P<0.001; median solid portion size, 8.4 mm vs. 4.6 mm, P<0.001; median solid
proportion, 52.8% vs. 36.8%, P=0.032). Lobulation and spiculation were more frequently
observed in invasive lesions (invasive vs. preinvasive lesions: lobulation, 118/283 vs. 5/26,
P=0.025; spiculation, 122/283 vs. 4/26, P=0.006). There were no significant differences in
age, sex, family history of lung cancer, presence of emphysema, nodule location, and nodule
count per scan (table 3).
Table 3 Comparison of clinical and radiological characteristics between differing pathologic
diagnoses in patients with part-solid nodules
Page 14 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Note.
—
Unless
other
wise
specifi
ed,
data
are
numbe
rs of
patient
s
(with
percen
tages
in
parent
heses).
aData are median (with interquartile range in parentheses).
AAH, atypical adenomatous hyperplasia; AIS, adenocarcinoma-in-situ; GGN, ground-glass
nodule; IPA, invasive pulmonary adenocarcinoma; MIA, minimally invasive adenocarcinoma
SSN risk stratification using the Brock and Lee models
Characteristics AAH and AIS
(n=26)
MIA and IPA
(n=283)
P-value
Age (year)a 62 (54, 68) 63 (56, 70) 0.257
Sex Male 7 (26.9) 122 (43.1) 0.109
Female 19 (73.1) 161 (56.9)
Family history of lung
cancer
Yes 1 (3.8) 9 (3.2) 0.590
No 25 (96.2) 274 (96.8)
Emphysema Yes 2 (7.7) 32 (11.3) 0.752
No 24 (92.3) 251 (88.7)
Nodule size (mm)a 13.6 (9.8,
16.6)
17.6 (13.3,
22.4)
<0.001
Solid portion size
(mm)a
4.6 (3.4, 5.8) 8.4 (5.2, 13.6) <0.001
Solid proportion (%)a 36.8 (25.1,
63.2)
52.8 (33.4,
66.0)
0.032
Location Upper lobe 12 (46.2) 181 (64.0) 0.073
Other lobes 14 (53.8) 102 (36.0)
Lobulation Yes 5 (19.2) 118 (41.7) 0.025
No 21 (80.8) 165 (58.3)
Spiculation Yes 4 (15.4) 122 (43.1) 0.006
No 22 (84.6) 161 (56.9)
Nodule count per scana 3 (2,8) 3 (1,4) 0.132
Page 15 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
For pGGNs, the AUC of the Brock model’s predicted probability for differentiating
invasive lesions from preinvasive lesions was 0.671 [95% confidence interval (CI): 0.571,
0.762] (table 4). A cutoff of 10%, suggested by the BTS, yielded a sensitivity, specificity,
accuracy, PPV, and NPV of 32.2%, 90.5%, 56.4%, 82.6%, and 48.7%, respectively. Another
cutoff of 4.29%, an optimal threshold based on the Youden index, provided a sensitivity,
specificity, accuracy, PPV, and NPV of 64.4%, 64.3%, 64.4%, 71.7%, and 56.3%. Brock
model for pGGNs showed poor calibration (P<0.001). A nodule size cutoff (10 mm)
suggested by Lee et al.7 was also applied to our study population. The resultant sensitivity,
specificity, accuracy, PPV, and NPV were 76.3%, 42.9%, 62.4%, 65.2%, and 56.3%,
respectively. There were no significant differences in diagnostic accuracy between the Brock
model and Lee criteria (Brock model cutoff 10% vs. nodule size cutoff 10mm, P=0.461;
Brock model cutoff 4.29% vs. nodule size cutoff 10 mm, P=0.832).
Table 4 Performance of each prediction model in diagnosing minimally invasive
adenocarcinoma and invasive adenocarcinoma for pure ground-glass nodules
Note.— Data are in percentages except for AUC. There were 42 preinvasive lesions and 59
invasive lesions (minimally invasive adenocarcinomas and invasive adenocarcinomas).
aA cutoff of 10% predicted probability was used as suggested by the British Thoracic Society.
bA cutoff of 4.29% predicted probability, the optimal threshold based on the Youden index,
was used.
Sensitivity Specificity Accuracy PPV NPV AUC
Brock model1a 32.2 90.5 56.4 82.6 48.7 0.671
(0.571, 0.762) Brock model2b 64.4 64.3 64.4 71.7 56.3
Lee model
(Nodule sizec)
76.3 42.9 62.4 65.2 56.3 -
Page 16 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
cA cutoff of 10 mm nodule size was used.
AUC, area under the receiver operating characteristic curve; NPV, negative predictive value;
PPV, positive predictive value
As for PSNs, the AUC of the Brock model was 0.746 (95% CI: 0.694, 0.794) for the
discrimination of invasive lesions from preinvasive lesions (table 5). A cutoff of 10% yielded
a sensitivity, specificity, accuracy, PPV, and NPV of 82.3%, 53.8%, 79.9%, 95.1%, and
21.9%. The optimal cutoff based on the Youden index was 10.11%, which was very close to
the suggested threshold by the BTS. Therefore, performance metrics were not calculated
separately. AUC of the Lee model was 0.771 (95% CI: 0.720, 0.817), and an optimal cutoff of
66.68% provided a sensitivity, specificity, accuracy, PPV, and NPV of 77.0%, 69.2%, 76.4%,
96.5%, and 21.7%, respectively. AUCs and diagnostic accuracies were not significantly
different between the two models (P=0.574 and 0.169, respectively). In addition, both models
exhibited poor calibration (P<0.001).
Table 5 Performance of each prediction model in diagnosing minimally invasive
adenocarcinoma and invasive adenocarcinoma for part-solid nodules
Note.— Data are in percentages except for AUC. There were 26 preinvasive lesions and 283
invasive lesions (minimally invasive adenocarcinomas and invasive adenocarcinomas).
aA cutoff of 10% predicted probability was used, which was suggested by the British
Thoracic Society and was optimal at the same time.
Sensitivity Specificity Accuracy PPV NPV AUC
Brock modela 82.3 53.8 79.9 95.1 21.9 0.746
(0.694, 0.794)
Lee modelb 77.0 69.2 76.4 96.5 21.7 0.771
(0.720, 0.817)
Page 17 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
bAn optimal cutoff of 66.68% was adopted based on the Youden index.
AUC, area under the receiver operating characteristic curve; NPV, negative predictive value;
PPV, positive predictive value
With respect to the pooled analysis for the entire SSNs, the AUC of Brock model was
0.810 (95% CI: 0.769, 0.846). Calibration was also poor in this model (P<0.001).
Page 18 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
DISCUSSION
In this study, we revealed that AUCs for the differentiation of invasive lesions among
PSNs using the established risk prediction models ranged from 0.746 to 0.771 with no
significant differences between the two models. Diagnostic accuracies based on the optimal
cutoffs were 79.9% for the Brock model and 76.4% for the Lee model. For pGGNs, the
diagnostic accuracy was 56.4%-64.4% for the Brock model depending on the cutoff values
used and 62.4% for the Lee criteria. For the entire SSNs, the Brock model showed AUC of
0.810.
McWilliams et al.16
originally developed a lung cancer prediction model (Brock
model) using participants enrolled in a lung cancer screening study. Thus, the Brock model
initially targeted pulmonary nodules detected on first screening CT. Incidentally detected
nodules as well as surgical candidates were not the original target lesions of this model.
However, at present, the BTS recommends using the same diagnostic approach for nodules
detected incidentally as those detected through screening.21
BTS also recommends using the
Brock model for the risk calculation of both solid and subsolid nodules.21
A cutoff of 10%
predicted probability for malignancy is suggested in order to differentiate high risk SSNs for
the performance of biopsy or surgical resection.21
This quantitative diagnostic approach is to
discern malignant SSNs with an appropriate false positive rate. However, it must be noted
that most persistent SSNs belong to one of the four categories of the adenocarcinoma
spectrum: AAH, AIS, MIA and IPA. Therefore, the potential of the risk prediction model in
discriminating lesions with invasive components (MIA and IPA) should also be tested using
pathologic diagnosis as a reference standard. Indeed, as clinical management strategies differ
substantially between preinvasive and invasive lesions, if it can be feasible to predict the
Page 19 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
invasiveness of SSNs, clinical planning whether to perform annual CT surveillance, limited
resection, or conventional lobectomy can be facilitated.
The performance of the prediction models for the risk stratification of SSNs was not
optimal according to our study results. For PSNs, AUCs of the two models ranged between
0.746 and 0.771 with diagnostic accuracies close to 80%. The performance of the prediction
model for PSNs in the original paper by Lee et al.7 was 0.905 (AUC). The study population
of the present study was similar to that of the study by Lee et al.7 An important reason for the
performance drop would be the spectrum effect, which is a common cause of model
performance heterogeneity.25
A variation in the assessment of CT morphologic features
(lobulation and spiculation) would be another potential cause. Past research on distinguishing
invasive adenocarcinomas appearing as SSNs have reported that logistic regression models
built with size metrics, morphologic features, or texture features showed AUCs ranging from
0.79 to 0.98.4-6 9 11 13
However, these models were not tested for an independent cohort or
validated externally.
Another important finding of our study was that the PPV for the differentiation of
invasive lesions among PSNs was very high for both models, over 95%. In other words, the
probability of being an invasive lesion was over 95% for nodules predicted as being invasive
through these models. A concern, however, is the high false negative rate of these models.
PSNs predicted as preinvasive lesions, which have a low calculated risk, should be managed
according to their solid portion size, if they are persistent lesions.26
A few studies have shown
that the solid portions in PSNs are well correlated to the pathologic invasive component.12 27
28 Fleischner Society guideline recommends that PSNs with solid components ≥6 mm be
monitored with CT scans at 3-6 months interval.26
PSNs with solid portions larger than 8 mm
should be biopsied or surgically resected in consideration of invasive adenocarcinomas.26
Page 20 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
BTS also recommends that the solid component size be considered to further refine the
estimate of malignancy risk.21
In addition, growing solid component is also a sign of an
invasive adenocarcinoma as described in both guidelines.21 26
The diagnostic accuracies of both the Brock model and Lee criteria were even lower
for pGGNs. Among multiple clinical and radiologic characteristics investigated in our study,
only three variables (family history of lung cancer, nodule size, and nodule count per scan)
were significantly different between preinvasive and invasive lesions. This implies the need
for other useful features for the development of new better prediction models. Features such
as nodule volume, mass, or radiomic features may provide additional clues for their
differentiation.4 29
In addition, changes in nodule characteristics at follow-up CT scans, such
as an increase in nodule size, attenuation, or new development of a solid portion, may also be
valuable for the discrimination.14
Alternatively, computational classification analysis
including deep learning algorithms, which do not require hand-crafted features and can be
self-trained directly from raw image pixels, may be another solution for the diagnosis of
pGGNs.30
The Brock model has been externally validated for the cohorts of the Danish Lung
Cancer Screening Trial (DLCST)19
and National Lung Screening Trial (NLST);18
AUCs for
the discrimination of malignant from benign nodules ranged from 0.834 for the former and
0.963 for the latter. AUCs for the validation cohort of the original paper, the British Columbia
Cancer Agency (BCCA) chemoprevention trial cohort, was 0.970.16
In addition, for an
Australian lung cancer screening cohort, Zhao et al.20
tested the utility of the Brock model for
the baseline evaluation of 52 SSNs and demonstrated that the AUC was 0.89. To the contrary,
however, the model performance evaluated in our study was lower than those reported in the
literature. The main reason for such a discrepancy may be that we included patients who
Page 21 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
underwent surgical resection of SSNs unlike previous studies. Thus, the proportion of
preinvasive lesions was small (16.6%) and a major portion of our study population consisted
of invasive lesions (83.4%). Such high prevalence of invasive lesions would have affected
our study results. Nevertheless, SSNs of interest in daily clinical practice may be closer to
those in our study. In routine practice, transient SSNs, which are definitely benign, do not
require risk calculation as they are easily confirmed through follow-up CT scans at short-term
intervals.26
In addition, small SSNs <6 mm are usually preinvasive and do not require CT
surveillance. On the other hand, particular concern should be given to persistent SSNs ≥6 mm,
especially to those with solid components. As the role of biopsy or positron emission
tomography is limited for SSNs,15
we supposed that risk prediction models may provide
value for more appropriate management planning. In this context, we applied prediction
models to surgically resected SSNs for the validation of their clinical utility.
There were several limitations to our study. First, our study was not conducted for a
screening cohort as described earlier in this manuscript. The prevalence of preinvasive lesions
was low compared to that of the screening setting. Thus, the performance measures in this
study should be carefully interpreted with respect to the target population, which were the
incidentally detected surgical candidates. Second, our retrospective study included a small
number of patients and analyses were conducted separately for pGGNs and PSNs. Separate
analysis of pGGNs and PSNs has resulted in a slight underestimation of the performance of
Brock model. Third, optimal cutoffs for the models were not obtained from ROC analyses of
the original study populations from which the models were derived. Fourth, radiologic nodule
information was extracted from our heterogeneous CT dataset, in which CT acquisition
parameters such as radiation dosage, slice thickness or contrast-enhancement were not
uniform across the study population. However, these factors would have had little effect on
Page 22 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
the variables we used. In addition, all CT scans had thin-section images (slice thickness ≤1.5
mm). Fifth, nodule size as well as solid portion size were measured as the longest transverse
diameter in accordance with the definition of lesion size and solid proportion in the original
papers. However, recent analyses have revealed that the usage of average diameter as an
input variable may enhance the model performance.31
In conclusion, the performance of the Brock model and Lee model for the
differentiation of invasive lesions among SSNs was suboptimal. In particular, both models
showed lower performance for pGGNs compared to that for PSNs. Thus, an alternative
approach such as computer-aided classification should be developed for the pre-operative
diagnosis of invasive lesions among SSNs.
Page 23 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Acknowledgements
We would like to thank Chris Woo, B.A., for editorial assistance.
Contributors
HK and CMP contributed to conception and design; HK, SJ, JHL, SYA, REY, HL, JP,
WHL, EJH, SML and JMG contributed to acquisition of data, or analysis and interpretation
of data; HK, CMP and JMG were involved in drafting the manuscript or revising it critically
for important intellectual content; all authors gave approval of the final version of the
manuscript.
Funding
This study was supported by Basic Science Research Program through the National
Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT & Future
Planning (grant number: 2017R1A2B4008517). Funder had no role in the study design; in the
collection, analysis and interpretation of the data; in the writing of the report; and in the
decision to submit the paper for publication.
Competing interests
None declared.
Patient consent
The requirement for written informed consent was waived by the Institutional
Review Board.
Page 24 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Ethics approval
Institutional Review Board of Seoul National University Hospital.
Data sharing statement
All data are available from the corresponding author.
Page 25 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
REFERENCES
1. Austin JH, Garg K, Aberle D, et al. Radiologic implications of the 2011 classification of
adenocarcinoma of the lung. Radiology 2013;266:62-71.
2. Yankelevitz DF, Yip R, Smith JP, et al. CT Screening for lung cancer: nonsolid nodules in
baseline and annual repeat rounds. Radiology 2015;277:555-64.
3. Scholten ET, de Jong PA, de Hoop B, et al. Towards a close computed tomography
monitoring approach for screen detected subsolid pulmonary nodules? Eur Respir J
2015;45:765-73.
4. Chae HD, Park CM, Park SJ, et al. Computerized texture analysis of persistent part-solid
ground-glass nodules: differentiation of preinvasive lesions from invasive pulmonary
adenocarcinomas. Radiology 2014;273:285-93.
5. Ding H, Shi J, Zhou X, et al. Value of CT characteristics in predicting invasiveness of
adenocarcinoma presented as pulmonary ground-glass nodules. Thorac Cardiovasc
Surg 2017;65:136-41.
6. Jin C, Cao J, Cai Y, et al. A nomogram for predicting the risk of invasive pulmonary
adenocarcinoma for patients with solitary peripheral subsolid nodules. J Thorac
Cardiovasc Surg 2017;153:462-69 e1.
7. Lee SM, Park CM, Goo JM, et al. Invasive pulmonary adenocarcinomas versus preinvasive
lesions appearing as ground-glass nodules: differentiation by using CT features.
Radiology 2013;268:265-73.
8. Li Q, Fan L, Cao ET, et al. Quantitative CT analysis of pulmonary pure ground-glass
nodule predicts histological invasiveness. Eur J Radiol 2017;89:67-71.
9. Liang J, Xu XQ, Xu H, et al. Using the CT features to differentiate invasive pulmonary
adenocarcinoma from pre-invasive lesion appearing as pure or mixed ground-glass
Page 26 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
nodules. Br J Radiol 2015;88:20140811.
10. Moon Y, Sung SW, Lee KY, et al. Pure ground-glass opacity on chest computed
tomography: predictive factors for invasive adenocarcinoma. J Thorac Dis
2016;8:1561-70.
11. Son JY, Lee HY, Kim JH, et al. Quantitative CT analysis of pulmonary ground-glass
opacity nodules for distinguishing invasive adenocarcinoma from non-invasive or
minimally invasive adenocarcinoma: the added value of using iodine mapping. Eur
Radiol 2016;26:43-54.
12. Yanagawa M, Johkoh T, Noguchi M, et al. Radiological prediction of tumor invasiveness
of lung adenocarcinoma on thin-section CT. Medicine (Baltimore) 2017;96:e6331.
13. Zhang Y, Shen Y, Qiang JW, et al. HRCT features distinguishing pre-invasive from
invasive pulmonary adenocarcinomas appearing as ground-glass nodules. Eur Radiol
2016;26:2921-8.
14. Kim H, Park CM, Koh JM, et al. Pulmonary subsolid nodules: what radiologists need to
know about the imaging features and management strategy. Diagn Interv Radiol
2014;20:47-57.
15. Naidich DP, Bankier AA, MacMahon H, et al. Recommendations for the management of
subsolid pulmonary nodules detected at CT: a statement from the Fleischner Society.
Radiology 2013;266:304-17.
16. McWilliams A, Tammemagi MC, Mayo JR, et al. Probability of cancer in pulmonary
nodules detected on first screening CT. N Engl J Med 2013;369:910-9.
17. Al-Ameri A, Malhotra P, Thygesen H, et al. Risk of malignancy in pulmonary nodules: a
validation study of four prediction models. Lung Cancer 2015;89:27-30.
18. White CS, Dharaiya E, Campbell E, et al. The Vancouver lung cancer risk prediction
model: assessment by using a subset of the National Lung Screening Trial cohort.
Page 27 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Radiology 2017;283:264-72.
19. Winkler Wille MM, van Riel SJ, Saghir Z, et al. Predictive accuracy of the PanCan lung
cancer risk prediction model -external validation based on CT from the Danish Lung
Cancer Screening Trial. Eur Radiol 2015;25:3093-9.
20. Zhao H, Marshall HM, Yang IA, et al. Screen-detected subsolid pulmonary nodules: long-
term follow-up and application of the PanCan lung cancer risk prediction model. Br J
Radiol 2016;89:20160016.
21. Callister ME, Baldwin DR, Akram AR, et al. British Thoracic Society guidelines for the
investigation and management of pulmonary nodules. Thorax 2015;70 Suppl 2:ii1-
ii54.
22. Travis WD, Brambilla E, Noguchi M, et al. International association for the study of lung
cancer/American thoracic society/European respiratory society international
multidisciplinary classification of lung adenocarcinoma. J Thorac Oncol 2011;6:244-
85.
23. Kim H, Park CM, Song YS, et al. Measurement variability of persistent pulmonary
subsolid nodules on same-day repeat CT: what is the threshold to determine true
nodule growth during follow-up? PLoS One 2016;11:e0148853.
24. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more
correlated receiver operating characteristic curves: a nonparametric approach.
Biometrics 1988;44:837-45.
25. Riley RD, Ensor J, Snell KI, et al. External validation of clinical prediction models using
big datasets from e-health records or IPD meta-analysis: opportunities and challenges.
BMJ 2016;353:i3140.
26. MacMahon H, Naidich DP, Goo JM, et al. Guidelines for management of incidental
pulmonary nodules detected on CT images: from the Fleischner Society 2017.
Page 28 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Radiology 2017;284:228-43.
27. Ko JP, Suh J, Ibidapo O, et al. Lung adenocarcinoma: correlation of quantitative CT
findings with pathologic findings. Radiology 2016;280:931-9.
28. Lee KH, Goo JM, Park SJ, et al. Correlation between the size of the solid component on
thin-section CT and the invasive component on pathology in small lung
adenocarcinomas manifesting as ground-glass nodules. J Thorac Oncol 2014;9:74-82.
29. Song YS, Park CM, Park SJ, et al. Volume and mass doubling times of persistent
pulmonary subsolid nodules detected in patients without known malignancy.
Radiology 2014;273:276-84.
30. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer
with deep neural networks. Nature 2017;542:115-18.
31. van Riel SJ, Ciompi F, Jacobs C, et al. Malignancy risk estimation of screen-detected
nodules at baseline CT: comparison of the PanCan model, Lung-RADS and NCCN
guidelines. Eur Radiol 2017;27:4019-29.
Page 29 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
FIGURE LEGENDS
Figure 1 Flow chart of patient inclusion and exclusion.
pGGN, pure ground-glass nodule; PSN, part-solid nodule; SSN, subsolid nodule
Page 30 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Flow chart of patient inclusion and exclusion.
127x157mm (600 x 600 DPI)
Page 31 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
CT SCANNING PROTOCOLS FOR THE STUDY POPULATION
Patients underwent CT scans using seven different scanners from four vendors. CT scans in our study were performed with the
following scanning protocols.
Vendor Toshibaa Siemens
b Siemens
b GE
c GE
c Philips
d Philips
d
Machine Aquilion One Definition Sensation
16
LightSpeed
Ultra
Discovery
CT750 HD
Brilliance 64 Ingenuity
kVp 120 120 120 120 120 120 120
mAs Target standard
deviation of
noise, 14
Quality
reference
mAs, 150
Quality
reference
mAs, 100
Noise index,
16.36
Noise
index, 22.67
Reference
mAs, 200
Reference
mAs, 170
Automatic
exposure
control
Sure Exposure CARE
Dose 4D
CARE
Dose 4D
Auto mA Auto mA Doseright
ACS, Z
DOM
Doseright
ACS, Z
DOM
Rotation time
(sec)
0.5 0.5 0.5 0.5 0.5 0.5 0.5
Collimation
(mm)
0.5x100 0.6x64 0.75x16 1.25x8 0.625x64 0.625x64 0.625x64
Slice thickness
(mm)
1 1 1 1 1.25 1 1
Pitch HP 103 1 1 0.875 0.984 1.014 1.0015
Page 32 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
Kernel FC 15-H B60f B60f Bone Chest YC 0 YC 0
Increment 1 1 1 1 1.25 1 1
Matrix 512x512 512x512 512x512 512x512 512x512 512x512 512x512
* All patients were scanned in supine position from the lung apex to the base during suspended maximum inspiration.
* For contrast enhancement, a total of 70-90 mL of 370 mgI/mL of the contrast material, iopamidol (Pamiray 370; Dongkook Pharmaceutical,
Seoul, Korea) or iopromide (Ultravist 370; Schering, Berlin, Germany), was injected at a rate of 2.0-2.3mL/sec using a power injector. CT
scanning was performed with a 60-second delay.
aToshiba Medical Systems, Otawara, Japan
bSiemens Healthcare, Forchheim, Germany
cGE Healthcare, Waukesha, WI, USA
dPhilips Healthcare, Cleveland, OH, USA
Page 33 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
TRIPOD Checklist: Prediction Model Validation
Section/Topic Item Checklist Item Page
Title and abstract
Title 1 Identify the study as developing and/or validating a multivariable prediction model, the target population, and the outcome to be predicted.
1
Abstract 2 Provide a summary of objectives, study design, setting, participants, sample size, predictors, outcome, statistical analysis, results, and conclusions.
3
Introduction
Background and objectives
3a Explain the medical context (including whether diagnostic or prognostic) and rationale for developing or validating the multivariable prediction model, including references to existing models.
5 (line 17), 6 (line 3)
3b Specify the objectives, including whether the study describes the development or validation of the model or both.
6 (line 7)
Methods
Source of data
4a Describe the study design or source of data (e.g., randomized trial, cohort, or registry data), separately for the development and validation data sets, if applicable.
7 (line 6-7)
4b Specify the key study dates, including start of accrual; end of accrual; and, if applicable, end of follow-up.
7 (line 7)
Participants
5a Specify key elements of the study setting (e.g., primary care, secondary care, general population) including number and location of centres.
2 (line 8), 9 (line 3)
5b Describe eligibility criteria for participants. 7
5c Give details of treatments received, if relevant. N/A
Outcome 6a
Clearly define the outcome that is predicted by the prediction model, including how and when assessed.
10 (line 2-3)
6b Report any actions to blind assessment of the outcome to be predicted. -
Predictors
7a Clearly define all predictors used in developing or validating the multivariable prediction model, including how and when they were measured.
9
7b Report any actions to blind assessment of predictors for the outcome and other predictors.
-
Sample size 8 Explain how the study size was arrived at. -
Missing data 9 Describe how missing data were handled (e.g., complete-case analysis, single imputation, multiple imputation) with details of any imputation method.
-
Statistical analysis methods
10c For validation, describe how the predictions were calculated. 10
10d Specify all measures used to assess model performance and, if relevant, to compare multiple models.
10-11
10e Describe any model updating (e.g., recalibration) arising from the validation, if done. N/A
Risk groups 11 Provide details on how risk groups were created, if done. N/A
Development vs. validation
12 For validation, identify any differences from the development data in setting, eligibility criteria, outcome, and predictors.
18, 21
Results
Participants
13a Describe the flow of participants through the study, including the number of participants with and without the outcome and, if applicable, a summary of the follow-up time. A diagram may be helpful.
12
13b Describe the characteristics of the participants (basic demographics, clinical features, available predictors), including the number of participants with missing data for predictors and outcome.
8
13c For validation, show a comparison with the development data of the distribution of important variables (demographics, predictors and outcome).
-
Model performance
16 Report performance measures (with CIs) for the prediction model. 15-17
Model-updating 17 If done, report the results from any model updating (i.e., model specification, model performance).
N/A
Discussion
Limitations 18 Discuss any limitations of the study (such as nonrepresentative sample, few events per predictor, missing data).
21
Interpretation
19a For validation, discuss the results with reference to performance in the development data, and any other validation data.
19-20
19b Give an overall interpretation of the results, considering objectives, limitations, results from similar studies, and other relevant evidence.
22
Implications 20 Discuss the potential clinical use of the model and implications for future research. 19
Other information
Page 34 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
For peer review only
TRIPOD Checklist: Prediction Model Validation
Supplementary information 21
Provide information about the availability of supplementary resources, such as study protocol, Web calculator, and data sets.
24 (line 2)
Funding 22 Give the source of funding and the role of the funders for the present study.
23 (line 9-11)
We recommend using the TRIPOD Checklist in conjunction with the TRIPOD Explanation and Elaboration document.
Page 35 of 35
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml
BMJ Open
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960
on February 5, 2020 by guest. P
rotected by copyright.http://bm
jopen.bmj.com
/B
MJ O
pen: first published as 10.1136/bmjopen-2017-019996 on 24 M
ay 2018. Dow
nloaded from
Top Related