Validation of the Surgical Outcome Risk Tool (SORT) in patients with pancreatic cancer undergoing surgery

Dear Editor,

Pancreatic resection is currently accepted as the mainstay of the multimodal treatment strategy for resectable and borderline pancreatic cancer. In this context, the evaluation of patients’ clinical status, along with the risk of perioperative morbidity and mortality for this type of major surgery is crucial to support the shared decision-making process, along with enhancing the oncologic treatment strategy, counseling, and outcome. The Surgical Outcome Risk Tool (SORT) was developed following the 2011 National Confidential Enquiry into Patient Outcome and Death (NCEPOD) report, in order to provide enhanced identification of high-risk surgical patients. ¹ To achieve this goal, SORT employs only six variables, designed to predict a patient's probability of 30-day postoperative mortality. Currently, it has been compared favorably with other risk stratification tools and has been externally validated in patients undergoing hip fracture and liver surgery. ^{2, 3} However, it has not been validated for a pancreatic cancer surgical population. The purpose of the present study was to validate the SORT model in Greek adult patients undergoing surgery for pancreatic cancer. We also compared SORT with two additional risk stratification tools, the Physiology and Operative Severity Score for the enumeration of Mortality and Morbidity (POSSUM), and the Portsmouth POSSUM (P-POSSUM).

Data were obtained from a prospectively maintained database of consecutive patients undergoing surgery for pancreatic cancer between January 1^st, 2017 and December 31^st, 2019 and ethical approval was obtained by the Scientific Committee of the University Hospital of Larissa, Greece (Protocol number: 50271/30-10-19). All the procedures were performed by the same surgical team leaded by the senior author (DZ). No imputation methods were used regarding missing data. We assessed the discrimination (i.e., the ability to separate those who died from those who did not die) and calibration (i.e., the ability to predict mortality rates in agreement with actual observed mortality rates) of the SORT model. Discrimination was assessed by generating Receiver-Operating Characteristic (ROC) curves and by calculating the Area Under the ROC Curve (AUC). The AUC was determined by calculating the 95% Confidence Intervals and compared using nonparametric paired tests, as described by DeLong et al. ⁴ We defined as poor, fair, and excellent model discrimination the AUC of < 0.70, 0.70-0.79 and 0.80-1.00, respectively. The calibration regarding each model was evaluated by estimating the predicted mortality (expected) and then comparing with the true mortality (observed). The observed/expected ratio of 1 represents perfect accuracy, a ratio < 1 indicates overprediction of mortality rate, and a ratio > 1 indicates underestimation. Calibration was further evaluated using the Hosmer-Lemeshow (H-L) goodness of fit test, defining a lack of fit as a p-value ≤ 0.05. ⁵ Finally, Chi-squared testing was used to compare the observed and expected outcome of all patients. All data were analyzed using Microsoft® Excel 16.36 (Microsoft, Redmond, Washington, USA) and Prism® Graphpad 8.4.2 for MacOS (GraphPad Software, San Diego, CA).

Fifty patients with pancreatic cancer were incorporated in the present analysis (Table 1), with a mean age of 66.7 years. The mean length of hospital stay was 17.52 (±7.29) days and the mean length of stay in the intensive care unit was 0.98 (±0.42) days. In the current study we reported a 30-day mortality rate of 6% (3 patients). SORT was associated with an excellent discrimination level (AUC = 0.96 [95% CI: 0.89-1.00]; p = 0.008). The ROC curve is demonstrated in Figure 1. SORT also demonstrated a significantly low H-L value (H-L = 0.02; p > 0.99), thus passing the goodness of fit test. Nonetheless, it underestimated the mortality rate (O:E = 1.5). POSSUM demonstrated a lower discrimination level (AUC = 0.89 [95% CI: 0.70-1.00]; p = 0.026) and a higher H-L value (H-L = 1.77; p = 0.99). It also underestimated mortality (O:E = 1.5). P-POSSUM was also associated with an excellent discrimination level (AUC = 0.95 [95% CI: 0.87-1.00]; p = 0.010), but lower than SORT, while underestimating the mortality rate at a higher level compared with SORT (O:E = 3). In addition, P-POSSUM was associated with a higher H-L value (H-L = 1.58; p = 0.99) in comparison to SORT.

Table 1
Patient baseline characteristics.

Demographics	Number of patients, n = 50
Female, n (%)	21 (42)
Mean age, years (SD)	66.6 (11)
ASA physical status , n (%)
I	10 (20)
II	23 (46)
III	15 (30)
IV	2 (4)
Operation priority, n (%)
Elective	50 (100)
Acute	0 (0)
Surgical operations, n (%)
Pancreaticoduodenectomy	42 (84)
Total pancreatectomy	1 (2)
Distal pancreatectomy	7 (14)
30-day mortality, n (%)	3 (6)

• ASA, American Society of Anesthesiologists.

Figure 1
The Receiver Operating Characteristics (ROC) Curve demonstrating the discrimination level of the Surgical Outcome Risk Tool (SORT) in patients with pancreatic cancer undergoing surgery.

There are certain limitations to the present study. In fact, the design of the study was retrospective, and the study population was small. Nonetheless, this is the first evidence regarding the validity of SORT in patients with pancreatic cancer undergoing surgery. In addition, we demonstrated that SORT is associated with excellent discrimination and an appropriate level of calibration in predicting postoperative mortality. Furthermore, our outcomes suggest the superiority of SORT compared with POSSUM and P-POSSUM. Future studies should further assess SORT in a greater study population of patients with pancreatic cancer undergoing surgery, with a greater follow-up, along with comparing it with other risk assessment tools.

Ethical approval

Ethical approval was obtained by the Scientific Committee of the University Hospital of Larissa (Protocol nº 50271/30-10-19).

• Ethical approval

  Ethical approval was obtained by the Scientific Committee of the University Hospital of Larissa (Protocol no 50271/30- 10-19).

References

• ¹
  Protopapa KL, Simpson JC, Smith NC, et al. Development, and validation of the Surgical Outcome Risk Tool (SORT). Br J Surg. 2014;101:1774-83.
• ²
  Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC-curve estimates obtained from partially paired datasets. Med Dec Making. 1998;18:110-21.
• ³
  Wong GTC, Ang WC, Wong TCL, et al. Surgical Outcome Risk Tool (SORT) validation in hepatectomy. Anaesthesia. 2017;72:1287-9.
• ⁴
  DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837-45.
• ⁵
  Hosmer DW, Hosmer T, Le Cessie S, et al. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med. 1997;16:965-80.

Publication Dates

• Publication in this collection
  30 June 2021
• Date of issue
  May-Jun 2021

History

• Received
  15 May 2020
• Accepted
  31 Oct 2020