Validation of APACHE IV score in postoperative liver transplantation in southern Brazil: a cohort study

Abstract

Background:

Liver transplantation is the only curative therapeutic modality available for individuals at end-stage liver disease. There is no reliable method of predicting the early postoperative outcome of these patients. The Acute Physiology and Chronic Health Evaluation (APACHE) is a widely used model for predicting hospital survival and benchmarking in critically ill patients. This study evaluated the calibration and discrimination of APACHE IV in the postoperative period of elective liver transplantation in the southern Brazil.

Methods:

This was a clinical prospective and unicentric cohort study that included 371 adult patients in the immediate postoperative period of elective liver transplantation from January 1, 2012 to December 31, 2016.

Results:

In this study, liver transplant patients who evolved to hospital death had a significantly higher APACHE IV score (82.7 ± 5.1 vs. 51.0 ± 15.8; p < 0.001) and higher predicted mortality (6.5% [4.4-20.2%] vs. 2.3% [1.4-3.5%]; p < 0.001). The APACHE IV score showed an adequate calibration (Hosmer-Lemeshow - H-L = 11.37; p = 0.181) and good discrimination (Receiver Operator Curve - ROC of 0.797; Confidence Interval 95% - 95% CI 0.713-0.881; p < 0.0001), although Standardized Mortality Ratio (SMR = 2.63), (95% CI 1.66-4.27; p < 0.001) underestimate mortality.

Conclusions:

In summary, the APACHE IV score showed an acceptable performance for predicting a hospital outcome in the postoperative period of elective liver transplant recipients.

KEYWORDS
Liver transplantation; APACHE IV; Validation

Resumo

Introdução:

O transplante de fígado é a única modalidade terapêutica curativa disponível para indivíduos com doença hepática terminal. Não há método confiável de prever o resultado pós-operatório imediato desses pacientes. A Avaliação da Gravidade da Doença Crônica e Aguda com bases Fisiológicas (APACHE) é um modelo amplamente usado para prever a sobrevida hospitalar e fazer a avaliação comparativa de pacientes criticamente enfermos. Este estudo avaliou a calibração e discriminação do APACHE IV no pós-operatório de transplante hepático eletivo no sul do Brasil.

Métodos:

Estudo clínico prospectivo de coorte em centro único que incluiu 371 pacientes adultos no pós-operatório imediato de transplante hepático eletivo de 1 de janeiro de 2012 a 31 de dezembro de 2016.

Resultados:

Neste estudo, pacientes com transplante hepático que evoluíram para óbito hospitalar obtiveram escore APACHE IV significativamente maior (82,7 ± 5,1 vs. 51,0 ± 15,8; p < 0,001) e mortalidade prevista mais alta (6,5% [4,4% -20,2%] vs 2,3% [1,4% -3,5%], p < 0,001). O escore APACHE IV mostrou uma calibração adequada (Hosmer-Lemeshow - H-L = 11,37; p = 0,181) e boa discriminação (Receiver Operator Curve - ROC de 0,797; intervalo de confiança de 95% - IC 95% 0,713-0,881; p < 0,0001), embora a taxa de mortalidade padronizada (Standardized Mortality Ratio - SMR = 2,63), (IC 95% 1,66-4,27; p < 0,001) subestime a mortalidade.

Conclusões:

Em resumo, o escore APACHE IV mostrou um desempenho aceitável para predizer um desfecho hospitalar no período pós-operatório de receptores eletivos de transplante hepático.

PALAVRAS-CHAVE
Transplante hepático; APACHE IV; Validação

Background

Liver transplantation is the only curative therapeutic modality available for individuals at end-stage liver disease.¹ In-hospital mortality in the postoperative period may reach 10% of transplanted patients.² There is no reliable method of predicting the early outcome of these patients.³ Acute Physiology and Chronic Health Evaluation (APACHE) is a widely used predictive predictor in critical care and is periodically updated from a predominantly North American database.⁴ Few studies have evaluated the performance of newer versions of APACHE in the postoperative period of hepatic transplant recipients. Keegan and colleagues (2009) evaluated the performance of APACHE III as a predictor of mortality in a cohort of 918 liver transplant patients performed between 1996 and 2008 with an observed mortality rate of 10.6%. In this study, APACHE III was not a suitable discriminator with an ROC curve of 0.65 for in-hospital mortality.⁵ Hu and co-workers (2013) in a retrospective cohort of 195 liver transplant recipients with an observed mortality rate of 13.8% compared the performance of Model for End-stage Liver Disease (MELD) to APACHE IV for this purpose. In this study, APACHE IV performed better than MELD with an ROC curve of 0.937 vs. 0.694.⁶

The main objective of this study is evaluated the calibration and discrimination of APACHE IV in the postoperative period of elective liver transplantation in the Southern Brazil. Our previous hypothesis was that APACHE IV could present discrimination and calibration adequate for the prediction of hospital outcome in the postoperative period of liver transplants.

Materials and methods

Study design

This was a clinical prospective and unicentric cohort study conducted in a tertiary hospital, 11-bed transplant Intensive Care Unit (ICU) in southern Brazil (Dom Vicente Scherer Hospital, Irmandade Santa Casa de Misericórdia de Porto Alegre). All of the patients receiving deceased donor organs and were enrolled in the immediate postoperative period at a transplant unit at the Dom Vicente Scherer Hospital. Patient data were entered on site using a software program (Sistema Epimed Monitor, Epimed Solutions, Rio de Janeiro, Brazil). A single researched colleted the data for the accomplishment of the score. There were no missing data for the calculation of the score in any of the patients included in the study. There were no losses in follow up. APACHE IV score consists in age, chronic health conditions and physiologic data, collected in the first 24 h of ICU admission.

Patients

All hospitalized patients ≥18 years were included in the immediate postoperative period from January 1, 2012 to December 31, 2016. Only the first admission to the ICU was considered for each patient. Living donor transplant patients were excluded. Also were excluded the transplants performed on an emergency basis due to acute liver failure. Patient data as a result of patient care and recorded in the medical record were collected prospectively, until hospital discharge.

Ethics

This study was approved by the Research Ethics Committee at Irmandade Santa Casa de Misericórdia de Porto Alegre (Plataforma Brasil CAAE No. 19687113.8.2002.5335). The need for informed consent was waived since no intervention was required and no individual data were expected to be disclosed.

Assessments

Only first ICU admission for each patient was used to predict hospital mortality within the same hospitalization. The APACHE IV score was calculated in the first 24 h after ICU admission. The adjusted probability of hospitalar death, according to the diagnostic category of APACHE IV, was also calculated.

Statistical analysis

The statistical analysis of the data was performed through the program Stata version 12.0 (StataCorp LP, College Station, Texas, USA). Descriptive statistics were used to describe the data, with calculation of mean, standard deviation, median and interquartile range, according to the distribution of variables. Student's t-test was used to evaluate the difference between means and the Mann-Whitney test to evaluate the distribution difference between medians, according to the normality of the distribution of the variables, evaluated by Komolgorov-Smirnov test. To assess the discrimination and ability to classify survivors and non-survivors in the hospital was plotted a ROC curve and calculated the respective Area Under Receiver Operator Characteristic (AUROC) curve and its Confidence Interval 95% (95% CI) according to the APACHE IV score. The discrimination was considered to be excellent, very good, good, moderate and poor with AUROC values of 0.9-0.99, 0.8-0.89, 0.7-0.79, 0.6-0.69 and <0.6, respectively. The quality of predictions was assessed by looking at the goodness-of-fit Hosmer-Lemeshow (HL) test, evaluating the degree of calibration (degree of agreement between the predicted and observed death probability) across all of the strata of probabilities of death. In this analysis an H-L close to the degree of freedom, as equal to the number of categories minus 2 and a significance level greater than 5% (p > 0.05) indicates a good calibration for the model. A calibration curve was constructed by plotting a linear relationship between observed and predicted outcomes. The Standardized Mortality Ratio (SMR) with their respective 95% CI was calculated by dividing the observed mortality rate by the predicted mortality rate. An SMR equal to 1.0 indicated that the number of observed deaths equaled that of the expected number of deaths; an SMR > 1.0 indicated the occurrence of a greater number of deaths than expected.

Results

A total of 371 consecutive patients with a hospital mortality of 12.1% were included. Table 1 shows the values of central tendency and dispersion for age, length of hospital stay and predicted mortality score in the hospital obtained in this population of liver transplant recipients. Patients who evolved to death had a higher APACHE IV score (82.7 ± 5.1 vs. 51.0 ± 15.8; p < 0.001), higher predicted mortality (6.5% [4.4%-20.2%] vs. 2.3% [1.4 - 3.5%]; p < 0.001) as well as a shorter ICU mean length of stay (3 days [QR 2-5] vs. 5 days [IQR 4-7]; p < 0.001) and hospital mean length of stay (12 days [IQR 4-30] vs. 23 days [IQR 19-31]; p < 0.001]. Fig. 1 shows the sensitivity and specificity analysis for APACHE IV represented by Area Under Receiver Operator Curve (AUROC) in patients submitted to liver transplantation with death outcome in the hospital. Discrimination of the APACHE IV model shows a good accuracy to predict in-hospital mortality in liver transplantation with a ROC curve of 0.797 (95% CI 0.713-0.881) p < 0.0001 (Fig. 1). In calibration, the APACHE IV model shows adequate performance for hospital mortality (H-L = 11.37; p = 0.181) (Table 2). The SMR showed that APACHE IV underestimated observed in-hospital mortality (SMR = 2.63; 95% CI 1.66-4.27; p < 0.001) (Fig. 2).

Table 1
Central tendency and dispersion values for age, length of hospital stay, and mortality predicted for outcomes of discharge and death in the hospital obtained in liver transplant patients.

	Hospitalar outcome
	Total sample	Discharge	Death	p-Value
Liver transplantation (n)	371	326	45
Age in years (mean ± SD)	57.1 ± 9.4	56.9 ± 9.3	58.3 ± 9.9	0.366
Hospital length of stay in days (median and IQR)	23 (18-31)	23 (19-31)	12 (4-30)	<0.001
ICU length of stay	5 (3-7)	5 (4-7)	3 (2-5)	<0.001
APACHE IV score (mean ± SD)	54.9 ± 21.5	51.0 ± 15.8	82.7 ± 5.1	<0.001
Predict mortality (%)	2.4 (1.5-4.2)	2.3 (1.4-3.5)	6.5 (4.4-20.2)	<0.001

• APACHE IV, Acute Physiology and Chronic Health Evaluation IV; ICU, Intensive Care Unit; IQR, interquartile range; SD, standard deviation.

Figure 1
Analysis of the sensitivity and specificity for APACHE IV represented by the ROC curve (AUROC) in patients undergoing liver transplantation (outcome of death in the hospital). Legend: 95% CI, Confidence Interval 95%; APACHE IV, Acute Physiology and Chronic Health Evaluation IV; AUROC, Area Under Receiver Operator Curve.

Table 2
Values of the sensitivity and specificity analysis for APACHE IV, represented by the ROC curve (Area Under Receiver Operator Curve - AUROC), Hosmer-Lemeshow test to analyze the degree of calibration and predictive characteristics for correct classification, according to the discharge and death in the hospital obtained in liver transplant patients.

Liver transplantation	ROC curve	HL test	Predictive characteristics
	AUROC (95% CI)	X2 (p-value)	Se (%) (95% CI)	Sp (%) (95% CI)	Cutoff	CC (%)
Hospitalar outcome	0.797 (0.713-0.881)	11.37 (0.181)	33.3 (20.0-49.0)	98.7 (96.9-99.7)	≥97	90.8

• HL, Hosmer-Lemeshow; AUROC, Area Under Receiver Operator Curve; 95% CI, Confidence Interval 95%; Se, sensitivity; Sp, specificity; CC, correct classification.

Figure 2
Calibration plot of predicted versus observed mortality in validation of APACHE IV score for hospital mortality in postoperative liver transplantation (solid diagonal line represents ideal calibration).

Discussion

As far as the authors are concerned, this is the first study to evaluate the performance of APACHE IV score in the liver transplant elective postoperative in Brazil. In this, study liver transplant patients who evolved to hospital death had a significantly higher APACHE IV value and higher predicted mortality. Confirming our previous hypothesis, APACHE IV presented adequate performance for the prediction of hospital outcome in the postoperative period of liver transplants.

External validation of prognostic scores may be problematic in a population admitted to ICU because of a specific diagnosis.⁷ In transplant patients, Simplified Acute Physiology Score III (SAPS) was statistically significantly mis-calibrated, had only moderate discrimination and underestimated hospital mortality.⁸ These findings are not unexpected. General prognostic models usually do not perform well in specific subgroups of patients because they may be under-represented in the developed cohort.⁹ For some specific diagnoses, a specific prognostic model may be an attractive alternative.⁷

Our results are consistent with the described by Hu et al.,⁶ a study in which APACHE IV presented excellent calibration and discrimination for prediction of in-hospital outcome in patients in the postoperative period of liver transplantation. The good performance of APACHE IV may be due to the weight of the physiological changes in the estimated mortality. Of course, the physiological changes contribute with 65.6% of the sum for predicting hospital death, while the contribution of the other risk factors is distributed among five characteristics.¹⁰ The good performance occurred despite of donors and intra-operative aspects, as well as some characteristics of the recipient that can contribute to the outcome and are not measured by the score.^11-13

However, for the purpose of benchmarking between different critical units, the SMR is usually used. In our study, a high SMR demonstrated that there was an underestimation of the predicted mortality.

Our study is a unicentric study that used an administrative database. These limit the external validity of our findings. The retrospective nature of the study prevented us from evaluating other variables that may be important in the prognosis of these patients.

Conclusions

The APACHE IV score showed an acceptable performance for predicting a hospital outcome in the postoperative period of liver transplant recipients. The SMR obtained through APACHE IV is not reliable as benchmarking in the elective postoperative of liver transplant.

• Funding
  The authors used their own resources and received only logistical support from RIPIMI (Institutional Network for Research and Innovation in Intensive Care Medicine).

Acknowledgements

This study was supported by RIPIMI (Institutional Network for Research and Innovation in Intensive Care Medicine)/Irmandade Santa Casa de Misericórdia de Porto Alegre/RS, Brazil.

References

• ¹
  Ahmed A, Keeffe EB. Current indications and contraindications for liver transplantation. Clin Liver Dis. 2007;11:227-47.
• ²
  Arabi Y, Abbasi A, Goraj R, et al. External validation of a modified model of Acute Physiology and Chronic Health Evaluation (APACHE) II for orthotopic liver transplant patients. Crit Care. 2002;6:245-50.
• ³
  Volk ML, Hernandez JC, Lok AS, et al. Modified Charlson Comorbidity Index for predicting survival after liver transplantation. Liver Transpl. 2007;13:1515-20.
• ⁴
  Salluh JIF, Soares M. ICU severity of illness scores: APACHE SAPS and MPM. Curr Op Crit Care. 2014;20:557-65.
• ⁵
  Keegan MT, Gali B, Findlay JY, et al. APACHE III outcome prediction in patients admitted to the intensive care unit after liver transplantation: a retrospective cohort study. BMC Surg. 2009;9:11.
• ⁶
  Hu Y, Zhang X, Liu Y, et al. APACHE IV is superior to MELD scoring system in predicting prognosis in patients after orthotopic liver transplantation. Clin Dev Immunol. 2013;2013:809847.
• ⁷
  Nassar AP, Malbouisson LMS, Moreno R. Evaluation of simplified acute physiology score 3 performance: a systematic review of external validation studies. Critic Care. 2014;18:R117.
• ⁸
  Oliveira VM, Brauner JS, Rodrigues-Filho E, et al. Is SAPS 3 better than APACHE II at predicting mortality in critically ill transplant patients?. Clinics. 2013;68:153-8.
• ⁹
  Moreno RP, Metnitz PG, Almeida E, et al. SAPS 3 - From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med. 2005;31:1345-55.
• ¹⁰
  Zimmerman JE, Kramer AA, McNair DS, et al. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006;34:1297-310.
• ¹¹
  Azevedo LD, Stucchi RS, Ataíde EC, et al. Assessment of causes of early death after twenty years of liver transplantation. Transplant Proc. 2013;45:1116-8.
• ¹²
  Freeman RB. Deceased donor risk factors influencing liver transplant outcome. Transplant Int. 2013;26:463-70.
• ¹³
  De Boer MT, Christensen MC, Asmussen M, et al. The impact of intraoperative transfusion of platelets and red blood cells on survival after liver transplantation. Anesth Analg. 2008;106:32-44.

Publication Dates

• Publication in this collection
  12 Aug 2019
• Date of issue
  May-Jun 2019

History

• Received
  28 Apr 2018
• Accepted
  18 Nov 2018
• Published
  15 Feb 2019