CLINICAL ARTICLEJ Neurosurg 128:942–947, 2018

Adverse perioperative events represent a significant burden to the US health care system,10 driving heightened emphasis on quality of care, morbidity,

and mortality. A key component of this effort is holding physicians accountable for adverse events by reporting on

their performance, as measured by a series of quality met-rics. Physicians and institutions are incentivized to deliver quality health care based on these various metrics since poor performance can result in negative payment adjust-ments by the Centers for Medicare and Medicaid Services

ABBREVIATIONS ACS = American College of Surgeons; CMS = Centers for Medicare and Medicaid Services; CPT = Current Procedural Terminology; NSQIP = National Surgical Quality Improvement Program; OR = operating room; PQRS = Physician Quality Reporting System; PR = prevalence ratio; SSI = surgical site infection; UTI = uri-nary tract infection; VTE = venous thromboembolism.SUBMITTED August 29, 2016. ACCEPTED November 17, 2016.INCLUDE WHEN CITING Published online April 28, 2017; DOI: 10.3171/2016.11.JNS161377.* Dr. Vaziri and Mr. Wilson contributed equally to this work.

Predictive performance of the American College of Surgeons universal risk calculator in neurosurgical patients*Sasha Vaziri, MD,1,2 Jacob Wilson, BS,2 Joseph Abbatematteo, PharmD,2 Paul Kubilis, MS,1,2 Saptarshi Chakraborty, MS,3 Khare Kshitij, PhD,3 and Daniel J. Hoh, MD1,2

1Department of Neurosurgery, 2University of Florida College of Medicine; and 3Department of Statistics, University of Florida College of Liberal Arts and Sciences, Gainesville, Florida

OBJECTIVE The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) universal Surgical Risk Calculator is an online decision-support tool that uses patient characteristics to estimate the risk of adverse postoperative events. Further validation of this risk calculator in the neurosurgical population is needed; therefore, the object of this study was to assess the predictive performance of the ACS NSQIP Surgical Risk Calculator in neurosurgical patients treated at a tertiary care center.METHODS A single-center retrospective review of 1006 neurosurgical patients treated in the period from September 2011 through December 2014 was performed. Individual patient characteristics were entered into the NSQIP calculator. Predicted complications were compared with actual occurrences identified through chart review and administrative qual-ity coding data. Statistical models were used to assess the predictive performance of risk scores. Traditionally, an ideal risk prediction model demonstrates good calibration and strong discrimination when comparing predicted and observed events.RESULTS The ACS NSQIP risk calculator demonstrated good calibration between predicted and observed risks of death (p = 0.102), surgical site infection (SSI; p = 0.099), and venous thromboembolism (VTE; p = 0.164) Alternatively, the risk calculator demonstrated a statistically significant lack of calibration between predicted and observed risk of pneumonia (p = 0.044), urinary tract infection (UTI; p < 0.001), return to the operating room (p < 0.001), and discharge to a rehabilitation or nursing facility (p < 0.001). The discriminative performance of the risk calculator was assessed using the c-statistic. Death (c-statistic 0.93), UTI (0.846), and pneumonia (0.862) demonstrated strong discriminative performance. Discharge to a rehabilitation facility or nursing home (c-statistic 0.794) and VTE (0.767) showed adequate discrimination. Return to the operating room (c-statistic 0.452) and SSI (0.556) demonstrated poor discriminative perfor-mance. The risk prediction model was both well calibrated and discriminative only for 30-day mortality.CONCLUSIONS This study illustrates the importance of validating universal risk calculators in specialty-specific surgi-cal populations. The ACS NSQIP Surgical Risk Calculator could be used as a decision-support tool for neurosurgical informed consent with respect to predicted mortality but was poorly predictive of other potential adverse events and clini-cal outcomes.https://thejns.org/doi/abs/10.3171/2016.11.JNS161377KEY WORDS quality improvement; prediction; neurosurgical; preoperative risk; surgical risk calculator; ACS; NSQIP

J Neurosurg Volume 128 • March 2018942 ©AANS 2018, except where prohibited by US copyright law

Unauthenticated | Downloaded 03/16/22 06:33 PM UTC

Performance of the ACS risk calculator in neurosurgical patients

J Neurosurg Volume 128 • March 2018 943

(CMS).15 Preoperative risk assessment calculators can serve as valuable instruments for determining candidates at high risk for increased morbidity and mortality. Identi-fying the likelihood of perioperative adverse events before surgery can facilitate shared decision making in either choosing an alternative lower-risk intervention or pro-ceeding with surgery albeit with a better understanding of the potential risk.9

The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) univer-sal Surgical Risk Calculator is a recently available open web-based tool for determining the likelihood of adverse perioperative events.2 The risk calculator algorithm was modeled from data gathered at 393 hospitals, in 1,414,006 patients, and based on 1557 types of procedures as defined by Current Procedural Terminology (CPT) codes. The risk calculator uses individual patient characteristics to estimate the percentage risk for a defined list of compli-cations. As such, the risk calculator was specifically de-signed as a risk-stratification tool for surgeons to use with patients as part of the informed consent process.2

Experience with surgical risk calculators across the spectrum of surgical specialties is limited, with no cur-rently validated neurosurgery-specific instrument. The ACS NSQIP Surgical Risk Calculator was created from a database that included 44,603 neurosurgical procedures, or 3.2% of the total number of procedures analyzed, and thus can be used to calculate the risk of perioperative events based on patient characteristics and the planned surgery CPT code.2 Several recent studies in other sur-gical disciplines have identified discrepancies when com-paring the predicted risk from the ACS NSQIP calculator and the actual complication occurrence in large patient se-ries.1,5,16,17 Therefore, our objective in the present study was to assess the predictive performance of the ACS NSQIP Surgical Risk Calculator for perioperative adverse events in neurosurgical patients treated at a single tertiary care, academic medical center.

MethodsData Collection

Institutional review board approval was obtained prior to initiating this retrospective review of neurosurgical patients treated at the University of Florida in the period from September 2011 through December 2014. Inclusion criteria were limited to patients either with a single neu-rosurgical CPT code or with two CPT codes in which the secondary CPT code indicated the use of the oper-ating microscope. The rationale for this approach relates to a limitation specific to the ACS NSQIP risk calculator, which allows only a single CPT code to be manually en-tered. For surgical procedures that involve multiple CPT codes, there is no method by which the ACS NSQIP risk calculator can determine cumulative risk across the sum total of procedure codes. The neurosurgical procedures reviewed are listed by CPT code in Supplemental Table 1.

Individual patient characteristics were obtained by re-view of the medical records and were manually entered into the ACS NSQIP risk calculator (for the Surgical Risk Calculator graphic user interface, see http://riskcalculator.

facs.org/). For the purposes of this study, “none” was se-lected for the question, “Are there other potential treat-ment options?” Moreover, to minimize bias, “Surgeon Adjustment of Risks” was not used as part of the patient risk assessment. Predicted quality outcomes data and es-timated risk for each perioperative adverse event were re-corded, including mortality, return to the operating room (OR) after initial surgery, discharge to a skilled nursing or rehabilitation facility, surgical site infection (SSI), venous thromboembolism (VTE), urinary tract infection (UTI), pneumonia, cardiac complications, and acute renal failure. Predicted quality outcomes and estimated risk as deter-mined by the ACS NSQIP risk calculator were then com-pared with actual observed outcomes and the occurrence of adverse events, as reported by an internal institutional administrative and quality reporting database.

Statistical MethodsStatistical software (R version 3.3.1, The R Foundation)

was used to evaluate the performance of the ACS NSQIP risk calculator predictions. Prevalence ratios were used to compare the frequency of predicted and observed out-comes in the population. We computed 95% (corrected) bootstrap confidence intervals using methods described by DiCiccio and Efron.4 If the confidence interval includes 1, it implies that there is insufficient statistical evidence to indicate a difference in the frequency of predicted and observed outcomes. Additionally, the performance of the risk calculator’s predictions was evaluated with respect to calibration (the average goodness of fit between observed outcomes and corresponding predicted risk scores) and discrimination (the tendency for cases to be associated with higher risk scores and noncases to be associated with lower risk scores).4,6,14,18 Traditionally, a risk prediction model performs well if the predicted risk scores are both well calibrated and capable of discriminating between cases and noncases.

CalibrationMethods described by Harrell were used to assess

calibration for each type of binary or categorical surgical outcome.6 Calibration for each model (corresponding to a single outcome) was assessed first by fitting logit-trans-formed risk scores to outcomes through a simple linear model while allowing the intercept and slope in the model to vary freely, and then by performing the likelihood ratio test (LRT) of the (null) hypothesis of perfect alignment (that is, slope = 1 and intercept = 0). A test statistic mea-sured the ability of the model to assign accurate probabili-ties of a perioperative event, with a p value > 0.05 indicat-ing that the number of observed events was (statistically) comparable to the number of events predicted by the ACS NSQIP risk calculator.

DiscriminationModel discrimination was assessed with concordance

(c-statistic), or the area under the receiver operating char-acteristic curve. The c-statistic is the number of patients that experience events within a classification threshold as compared with the number of patients that do not experi-

Unauthenticated | Downloaded 03/16/22 06:33 PM UTC

S. Vaziri et al.

J Neurosurg Volume 128 • March 2018944

ence events within the same classification threshold.13 The c-statistic ranges from 0.5 (no discriminative value) to 1 (perfect discrimination), with a c-statistic of 0.5 indicating a discriminative value equivalent to a coin toss. Discrimi-nation is considered adequate when the c-statistic exceeds 0.7 and strong when the c-statistic exceeds 0.8.8,13

ResultsFrom September 1, 2011, to December 31, 2014, 1006

neurosurgical patients and procedures were reviewed (Supplemental Table 1). Demographic data and patient risk factors (Table 1) were entered into the ACS NSQIP risk calculator, and predicted complications were com-pared with the observed number of events (Table 2).

Prevalence ResultsPrevalence ratios (PRs) with 95% confidence intervals

(CIs) were used to compare the frequency of predicted outcomes to observed outcomes in the study population. Mortality (PR 0.81, 95% CI 0.614–1.122), return to the OR (0.908, 0.731–1.175), SSI (0.689, 0.475–1.07), and VTE (1.574, 0.946–2.889) had prevalence ratios approaching 1 with insufficient statistical evidence to indicate a dif-ference in the frequency of predicted and observed out-comes. Alternatively, discharge to a rehabilitation or nurs-ing facility (PR 0.683, 95% CI 0.623–0.752), UTI (8.033, 3.012–25.42), and pneumonia (1.824, 1.072–3.33) showed statistically significant differences in the frequency of predicted to observed events. Cardiac complications and renal failure were not included in the analysis given a lack of observed events.

Calibration ResultsCalibration assesses the ability of a risk prediction

model to match the number of actual events across deciles of risk-stratified subgroups. A p < 0.05 signifies that the null hypothesis—that actual observed events occurred at a frequency similar to events predicted by the risk calcu-lator—is false. In other words, a p < 0.05 indicates poor calibration of the risk calculator or a lack of fit between the two models. Mortality (p = 0.102), SSI (p = 0.099), and VTE (p = 0.164) demonstrated good calibration between the observed and predicted models. Alternatively, the risk calculator demonstrated a statistically significant lack of calibration between predicted and observed risk for return to the OR (p < 0.001), pneumonia (p = 0.044), UTI (p <

CONTINUED IN NEXT COLUMN »

TABLE 1. Baseline characteristics in 1006 neurosurgical patients treated at a single center

Variable No. %

Sex Female 546 54.274 Male 460 45.726Admission status Emergency 383 38.072 Routine elective 570 56.660 Trauma center 8 0.795 Unknown 15 1.491 Urgent 30 2.982Readmission 119 11.829Classified as emergency 191 18.986Assessed in emergency department 210 20.875Functional status Independent 943 93.738 Partially dependent 49 4.871 Totally dependent 13 1.292 Unknown 1 0.099ASA Physical Status Classification I 21 2.087 II 254 25.249 III 525 52.187 IV 190 18.887 V 16 1.590Wound class Unknown 6 0.596 Clean 884 87.873 Clean-contaminated 53 5.268 Contaminated 24 2.386 Dirty-infected 34 3.380 Not documented 5 0.497Steroid use for chronic condition 76 7.555Ascites w/in 30 days 1 0.099Sepsis 48 hrs prior to surgery Sepsis 3 0.298 Septic thrombophlebitis 1 0.099 SIRS 1 0.099Ventilator dependent 93 9.244Disseminated cancer 73 7.256Diabetes mellitus Insulin 60 5.964 Oral 92 9.145Use of hypertensive medications 478 47.515Previous cardiac event 118 11.730Congestive heart failure 11 1.093Dyspnea Any 1 0.099 Exertion 63 6.262 At rest 11 1.093Smoker 259 25.746

TABLE 1. Baseline characteristics in 1006 neurosurgical patients treated at a single center

Variable No. %

Chronic obstructive pulmonary disease 55 5.467Dialysis 6 0.596Acute renal failure 6 0.596

ASA = American Society of Anesthesiologists; SIRS = systemic inflammatory response syndrome.

» CONTINUED FROM PREVIOUS COLUMN

Unauthenticated | Downloaded 03/16/22 06:33 PM UTC

Performance of the ACS risk calculator in neurosurgical patients

J Neurosurg Volume 128 • March 2018 945

0.001), and discharge to a rehabilitation or nursing facility (p < 0.001).

Discrimination ResultsDiscrimination assesses the ability of the risk calcu-

lator to distinguish between cases with actual events and noncases. Discriminative performance was assessed us-ing the c-statistic. A c-statistic of 0.5 indicates random occurrence, whereas a c-statistic > 0.7 signifies adequate discrimination and > 0.8 indicates strong discrimination. Mortality (c-statistic 0.93), UTI (0.846), and pneumonia (0.862) demonstrated strong discriminative performance. Discharge to a rehabilitation facility or nursing home (c-statistic 0.794) and VTE (0.767) had adequate discrimina-tive performance. Return to the OR (c-statistic 0.452) and SSI (0.556) demonstrated poor model discrimination.

Traditionally, an ideal risk prediction model demon-strates good calibration and discrimination.8,13 The ACS NSQIP risk calculator performed well on both criteria only for predicted mortality. Otherwise, the risk calcu-lator performed variably with respect to calibration and discrimination among the remaining predicted outcome variables.

DiscussionIdeally, surgical risk calculators are a patient-centered

tool for assessing the risk of perioperative complications, facilitating shared decision making, and ultimately im-proving the quality of health care delivery. The CMS has identified surgical risk calculators as an important element in assessing physician performance with respect to the quality of health care delivery. The CMS Physician Qual-ity Reporting System (PQRS) program began in 2007 by providing financial incentives to participating health care providers. In 2013, the PQRS began introducing financial penalties for lack of participation. As of 2015, the PQRS stopped providing incentive payments and began apply-ing penalties to the total Medicare Part B fee-for-service amount. To avoid financial penalties, participating health care practitioners report 9 quality measures to the PQRS. Measure #358, which examines the percentage of patients

whose personalized risks of postoperative complications were assessed by a surgical team, is one of the quality metrics surgeons have the option of reporting. This mea-sure requires documenting the specific calculator used and communicating the calculated risk to the patient and his or her family.12 Moving forward in the current health care quality era, one can expect surgical risk calculators to become a routine part of any surgical evaluation.

The ACS NSQIP Surgical Risk Calculator was devel-oped to provide a risk-stratification tool based on an indi-vidual patient’s characteristics and a given surgical pro-cedure. The advantage of this risk calculator is its open web-based format and easy user interface. It is currently the only publicly available risk calculator that includes the spectrum of neurosurgical CPT codes and a breadth of postoperative complications. Further, the risk calculator was created from a large database of patients and proce-dures across multiple surgeons, hospitals, geographic re-gions, and payer statuses, with the expectation that this would enhance its external validity. Since its inception, the ACS NSQIP risk calculator’s predictive performance has been tested by several surgical specialties with vary-ing results. Edelstein et al. studied predicted versus ob-served adverse events for total knee and total hip arthro-plasty in a series of 1066 patients from publicly reported Medicare databases.5 These investigators determined that complication rates predicted by the ACS NSQIP calcula-tor correlated poorly with actual complication rates. In another study, Samson et al. assessed whether the ACS NSQIP risk calculator effectively predicted risk as a way to determine whether a patient with early stage non–small cell lung cancer was a good candidate for lobectomy or wedge resection or should instead undergo radiotherapy due to surgical risk.17 These researchers concluded that the risk calculator did not adequately risk-stratify these patients into surgical and nonsurgical groups. Moreover, the calculator underestimated the number of complica-tions in surgical patients probably because of the calcula-tor’s inability to account for disease-specific risk factors. Further illustrating this disagreement between predicted and actual values is a study performed by Arce et al., in which the authors determined that the ACS NSQIP surgi-

TABLE 2. Prevalence, c-statistic, and calibration scores for the ACS risk prediction model

OutcomePrevalence (%) PR (predicted/

observed)95% CI for PR

c-StatisticCalibration Curve

Observed Predicted Lower Upper Intercept Slope p Value

Mortality 3.479 2.817 0.81 0.614 1.122 0.93 0.869 1.269 0.102Return to OR 7.058 6.411 0.908 0.731 1.175 0.452 −2.963 −0.125 <0.001Discharge to facility 28.429 19.418 0.683 0.623 0.752 0.794 0.22 0.629 <0.001SSI 2.286 1.576 0.689 0.475 1.07 0.556 −1.67 0.495 0.099VTE 1.292 2.034 1.574 0.946 2.889 0.767 0.361 1.242 0.164UTI 0.298 2.396 8.033 3.012 25.42 0.846 −0.532 1.5 <0.001Pneumonia 1.292 2.357 1.824 1.072 3.33 0.862 −0.549 1.032 0.044Cardiac complication* 0 0.889 NA NA NA NA NA NA NARenal failure* 0 0.457 NA NA NA NA NA NA NA

NA = not applicable.* Not included in analysis given a lack of observed events.

Unauthenticated | Downloaded 03/16/22 06:33 PM UTC

S. Vaziri et al.

J Neurosurg Volume 128 • March 2018946

cal calculator did not accurately predict 30-day complica-tions in all outcome measures in patients who underwent microvascular head and neck reconstruction.1 Last, in a gynecology-oncology study of the predictive performance of the ACS NSQIP surgical calculator in patients undergo-ing laparotomy, Rivard et al. concluded that the surgical risk calculator performed well at predicting 30-day mor-tality and cardiac complications but was not accurate in predicting more common complications such as SSI, UTI, and so forth.16

Comparing the ACS NSQIP risk calculator with actual events in our large series of neurosurgical patients, we found that the risk calculator predicted a similar preva-lence for several actually observed outcomes. However, the calculator proved to be an effective risk prediction model with excellent calibration and discrimination only for mortality and not any other outcome measure. A re-cent study by the ACS indicated that the ACS NSQIP risk calculator as originally designed may require further re-calibration to refine its predictive performance as there is a tendency to overestimate risk for low- and high-risk patients and underestimate surgical risk for mid-risk pa-tients.11

There are several possible explanations for the ob-served differences between the ACS NSQIP–predicted outcomes and actual outcomes in our patient series. As previously stated, the risk calculator is designed to predict risk based on a single entered CPT code. This assumes that the risk for a given procedure can only be defined by a single CPT code. In developing the risk calculator, the ACS NSQIP database would have needed to capture a large enough and varied enough sample for that single CPT code to create an adequate risk prediction model. Many common neurosurgical procedures involve multiple CPT codes. The method by which the ACS NSQIP data-base and risk calculator accounted for this aspect of most neurosurgical procedures is not entirely clear. Further, uncertainty as to which CPT code to enter into the risk calculator (for surgeries involving multiple CPT codes) to best predict actual risk could limit its widespread general application. Developing a risk calculator that captures the cumulative risk from multiple CPT codes can improve the accuracy of the calculator in neurosurgical patients, espe-cially neurosurgical spine patients.

Additionally, the neurosurgical patient population in our study may not be representative of those included in the ACS NSQIP database used to design the risk calcula-tor. The database collected patient information across a variety of patient care settings and payer statuses.2 The neurosurgical patients included in the present study were from a single tertiary care, academic medical center in which nearly two-thirds identified as self-pay, Medicare, or Medicaid. Studies have shown that certain quality met-rics such as patient safety indicators, hospital-acquired conditions, hospital length of stay, and mortality rates correlate with payer status—with self-pay, Medicare, and Medicaid patients generally having worse outcomes than privately insured patients.7 The extent to which the higher proportion of non–privately insured patients in our study population impacted actual versus predicted risk is uncer-tain. Institutions that similarly treat a high volume of self-

pay and government-funded patients may also encounter discrepancies in the predictive performance of the ACS NSQIP risk calculator.

As quality improvement initiatives gain momentum across institutions, the positive impact of successful pro-grams may result in better outcomes than those predicted by the risk calculator. At the University of Florida, an ag-gressive neurosurgery department–wide VTE prophylaxis protocol was instituted and has demonstrated a significant reduction in the occurrence of deep venous thrombosis as compared with before initiating the protocol.3 When as-sessing the predictive performance of the ACS NSQIP risk calculator, we observed that the risk calculator over-estimated the prevalence of VTE compared with its actual occurrence in our surgically treated population. Similarly, a UTI reduction quality improvement program was imple-mented at our institution’s neurointensive care unit with beneficial results.19 As expected, our study population demonstrated a lower actual UTI rate than that predicted by the risk calculator.

A limitation of this study is that the ACS NSQIP risk calculator, which was derived from prospective patient data, was tested against a retrospective review of neuro-surgical patient records at a single institution. Inadequate patient history-taking and documentation in the medical record may have impacted the accuracy of the risk calcu-lator’s predictions based on individual patient characteris-tics. However, one could argue that the real-world practi-cal application of the risk calculator would be subject to these same potential documentation errors in everyday pa-tient-physician encounters. Further, actual adverse events for this study were obtained from an institutional internal quality improvement database, which prospectively re-cords individual patient complications. The potential for missed or underreported adverse events in this institution-al database may have also affected the discrepancy in ac-tual versus predicted risk.

ConclusionsThe ACS NSQIP Surgical Risk Calculator is a tool

designed to assist both physicians and patients in preop-erative shared decision making, with the goal of improv-ing overall quality of health care delivery by reducing preventable morbidity and mortality. Overall, the ACS NSQIP risk calculator performed well with regard to pre-dicting mortality in a retrospective study of neurosurgi-cal patients but was poorly predictive of all other potential adverse events and clinical outcomes. Several factors may have impacted these findings with further validation and refinement of the risk calculator needed specifically for neurosurgical patients. Ultimately, current use of the ACS NSQIP risk calculator may be effective only for counsel-ing high-risk patients with regard to potential periopera-tive mortality.

References 1. Arce K, Moore EJ, Lohse CM, Reiland MD, Yetzer JG, Et-

tinger KS: The American College of Surgeons National Surgical Quality Improvement Program surgical risk calcula-tor does not accurately predict risk of 30-day complications

Unauthenticated | Downloaded 03/16/22 06:33 PM UTC

Performance of the ACS risk calculator in neurosurgical patients

J Neurosurg Volume 128 • March 2018 947

among patients undergoing microvascular head and neck reconstruction. J Oral Maxillofac Surg 74:1850–1858, 2016

2. Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY, et al: Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 217:833–842, 842.e1–842.e3, 2013

3. Cox JB, Weaver KJ, Neal DW, Jacob RP, Hoh DJ: Decreased incidence of venous thromboembolism after spine surgery with early multimodal prophylaxis: Clinical article. J Neuro-surg Spine 21:677–684, 2014

4. DiCiccio TJ, Efron B: Bootstrap confidence intervals. Stat Sci 11:189–228, 1996

5. Edelstein AI, Kwasny MJ, Suleiman LI, Khakhkhar RH, Moore MA, Beal MD, et al: Can the American College of Surgeons risk calculator predict 30-day complications after knee and hip arthroplasty? J Arthroplasty 30 (9 Sup-pl):5–10, 2015

6. Harrell F: Regression Modeling Strategies. New York: Springer, 2001

7. Hooten KG, Neal D, Lovaton Espadin RE, Gil JN, Azari H, Rahman M: Insurance status influences the rates of report-able quality metrics in brain tumor patients: a nationwide inpatient sample study. Neurosurgery 76:239–248, 2015

8. Hosmer DW, Lemeshow S: Applied Logistic Regression, ed 3. Hoboken, NJ: Wiley, 2013

9. Knops AM, Legemate DA, Goossens A, Bossuyt PM, Ubbink DT: Decision aids for patients facing a surgical treatment decision: a systematic review and meta-analysis. Ann Surg 257:860–866, 2013

10. Lawson EH, Hall BL, Louie R, Ettner SL, Zingmond DS, Han L, et al: Association between occurrence of a postoper-ative complication and readmission: implications for quality improvement and cost savings. Ann Surg 258:10–18, 2013

11. Liu Y, Cohen ME, Hall BL, Ko CY, Bilimoria KY: Evalua-tion and enhancement of calibration in the American College of Surgeons NSQIP surgical risk calculator. J Am Coll Surg 223:231–239, 2016

12. McNiff K: Physician quality reporting system enters penalty phase: practices urged to prepare now to avoid Medicare pay-ment reductions. J Oncol Pract 8:363–364, 2012

13. Merkow RP, Hall BL, Cohen ME, Dimick JB, Wang E, Chow WB, et al: Relevance of the c-statistic when evaluating risk-adjustment models in surgery. J Am Coll Surg 214:822–830, 2012

14. Nagelkerke NJD: A note on a general definition of the coef-ficient of determination. Biometrika 78:691–692, 1991

15. Pope GC, Kautter J, Ellis RP, Ash AS, Ayanian JZ, Lezzoni LI, et al: Risk adjustment of Medicare capitation payments using the CMS-HCC model. Health Care Financ Rev 25:119–141, 2004

16. Rivard C, Nahum R, Slagle E, Duininck M, Isaksson Vogel R, Teoh D: Evaluation of the performance of the ACS NSQIP

surgical risk calculator in gynecologic oncology patients un-dergoing laparotomy. Gynecol Oncol 141:281–286, 2016

17. Samson P, Robinson CG, Bradley J, Lee A, Broderick S, Kreisel D, et al: The National Surgical Quality Improvement Program risk calculator does not adequately stratify risk for patients with clinical stage I non-small cell lung cancer. J Thorac Cardiovasc Surg 151:697–705, 705.e1, 2016

18. Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijke-mans MJ, Habbema JD: Validation and updating of predic-tive logistic regression models: a study on sample size and shrinkage. Stat Med 23:2567–2586, 2004

19. Titsworth WL, Hester J, Correia T, Reed R, Williams M, Guin P, et al: Reduction of catheter-associated urinary tract infections among patients in a neurological intensive care unit: a single institution’s success. J Neurosurg 116:911–920, 2012

DisclosuresThe authors report no conflict of interest concerning the materi-als or methods used in this study or the findings specified in this paper.

Author ContributionsConception and design: Hoh. Acquisition of data: Vaziri, Wilson, Abbatematteo. Analysis and interpretation of data: all authors. Drafting the article: Vaziri, Wilson, Chakraborty, Hoh. Critically revising the article: all authors. Reviewed submitted version of manuscript: all authors. Approved the final version of the manu-script on behalf of all authors: Vaziri. Statistical analysis: Kubilis, Chakraborty. Administrative/technical/material support: Kubilis, Hoh. Study supervision: Hoh.

Supplemental InformationOnline-Only ContentSupplemental material is available with the online version of the article.

Supplemental Table 1. https://thejns.org/doi/suppl/10.3171/ 2016. 11.JNS161377.

Previous PresentationsThis work was presented as an oral presentation at the American Association of Neurological Surgeons Annual Meeting held in Chicago, Illinois, on April 30–May 4, 2016.

CorrespondenceSasha Vaziri, Department of Neurosurgery, University of Florida, PO Box 100265, Gainesville, FL 32610-0261. email: sasha.vaziri@neurosurgery.ufl.edu.

Unauthenticated | Downloaded 03/16/22 06:33 PM UTC