SciELO - Scientific Electronic Library Online

vol.68 issue2Mineralocorticoid replacement during infancy for salt wasting congenital adrenal hyperplasia due to 21-hydroxylase deficiencySex hormones in Malay and Chinese men in Malaysia: are there age and race differences? author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand




Related links



Print version ISSN 1807-5932

Clinics vol.68 no.2 São Paulo  2013 



Is SAPS 3 better than APACHE II at predicting mortality in critically ill transplant patients?



Vanessa M. de OliveiraI, II; Janete S. Brauner,I, II, III; Edison Rodrigues FilhoII; Ruth G. A. SusinI,II; Viviane DraghettiI, II; Simone T. BolzanI, II; Silvia R. R. VieiraI, IV

IHospital de Clínicas de Porto Alegre, Intensive Care Unit, Porto Alegre/RS, Brazil
IIHospital Dom Vicente Scherer, Intensive Care Unit, Porto Alegre/RS,Brazil
IIIInternal Medicine Unit, Hospital N. Senhora da Conceicao, Porto Alegre/RS, Brazil
IV Universidade Federal do Rio Grande do Sul, Porto Alegre/RS, Brazil




OBJECTIVES: This study compared the accuracy of the Simplified Acute Physiology Score 3 with that of Acute Physiology and Chronic Health Evaluation II at predicting hospital mortality in patients from a transplant intensive care unit.
METHOD: A total of 501 patients were enrolled in the study (152 liver transplants, 271 kidney transplants, 54 lung transplants, 24 kidney-pancreas transplants) between May 2006 and January 2007. The Simplified Acute Physiology Score 3 was calculated using the global equation (customized for South America) and the Acute Physiology and Chronic Health Evaluation II score; the scores were calculated within 24 hours of admission. A receiver-operating characteristic curve was generated, and the area under the receiver-operating characteristic curve was calculated to identify the patients at the greatest risk of death according to Simplified Acute Physiology Score 3 and Acute Physiology and Chronic Health Evaluation II scores. The Hosmer-Lemeshow goodness-of-fit test was used for statistically significant results and indicated a difference in performance over deciles. The standardized mortality ratio was used to estimate the overall model performance.
RESULTS: The ability of both scores to predict hospital mortality was poor in the liver and renal transplant groups and average in the lung transplant group (area under the receiver-operating characteristic curve = 0.696 for Simplified Acute Physiology Score 3 and 0.670 for Acute Physiology and Chronic Health Evaluation II). The calibration of both scores was poor, even after customizing the Simplified Acute Physiology Score 3 score for South America.
CONCLUSIONS: The low predictive accuracy of the Simplified Acute Physiology Score 3 and Acute Physiology and Chronic Health Evaluation II scores does not warrant the use of these scores in critically ill transplant patients.

Keywords: APACHE; SAPS; Kidney Transplantation; Liver Transplantation; Lung Transplantation; Critical Care Patients.




Scoring systems, which predict patient outcomes in intensive care units, have been studied for more than three decades. The performance of prognostic models encompasses two objective measurements: discrimination (the model's ability to classify survivors and non-survivors) and calibration (agreement between the observed and expected numbers of survivors and non-survivors) (1-8).

Acute Physiology and Chronic Health Evaluation (APACHE) scores and Simplified Acute Physiology Score (SAPS) models were previously validated as severity scoring systems. However, several groups were not well represented in the databases that were used to validate these prognostic systems, including transplant patients.

The validation of APACHE II included only 47 kidney transplant patients, without any lung or liver transplant recipients (9). The validation of APACHE III, a further update, involved 17,440 patients but included only 40 liver transplant patients; lung and kidney transplant patients were absent (10). The validation of SAPS 3 included 172 transplant patients: 90 liver, 55 kidney, eight lung, five kidney-pancreas, three pancreas and 11 cardiac transplant patients (11,12). For the validation of APACHE IV, 224 kidney and 139 liver transplant patients were enrolled, but no lung transplant patients were included (13).

SAPS 3 evaluates the data obtained during the first hour of intensive care unit (ICU) admission, which is useful for assessing a patient's condition upon arrival and eliminating the factors attributable to the level of the care unit. With these adjustments, there have been improvements in the prediction of early and late mortality, as well as increased score sensitivity.

We chose to validate the APACHE II scoring system instead of the APACHE IV because the former has been extensively used to assess severity of illness in transplant patients (14-16). However, to our knowledge, there have been no studies applying SAPS 3 specifically to transplant patients or comparing the predictive accuracy of the two scores in this patient group.

The objective of the present study was to compare the accuracy of SAPS 3 with that of APACHE II in predicting in-hospital mortality in patients from a transplant ICU.



Study design

This was a prospective clinical cohort study conducted in an 11-bed transplant ICU in southern Brazil (Dom Vicente Scherer Hospital, Complexo Hospitalar da Santa Casa de Porto Alegre). All of the patients receiving cadaver donor organs and admitted to the ICU between May 2006 and January 2007 were enrolled.


Patients aged >18-years after their first transplant were included in the study during the early postoperative period (<24 hours). Retransplant and living donor transplant patients were excluded.


This study was approved by the Research Ethics Committee of the institution and was conducted in accordance with the provisions of the Declaration of Helsinki. All of the participants or their family members signed informed consent forms. The data were collected and recorded by a single author.


The first ICU admission for each patient was used to predict hospital mortality. The APACHE II score was calculated in the first 24 hours after ICU admission, and the SAPS 3 score was calculated within 1 hour of admission (11,12).

The adjusted probability of death, according to the diagnostic category of APACHE II, was also calculated (9).

The probability of in-hospital death for SAPS 3 was calculated according to the general equation and was customized for South America. Customization has been more accurate in predicting the mortality of oncological patients admitted to ICUs in Brazil (11,12,17,18).

Statistical analysis

The data were analyzed using the Statistical Package for the Social Sciences (SPSS), version 13.0 for Windows (SPSS Inc. Chicago, IL, USA). Receiver-operating characteristic (ROC) curves were generated for the SAPS 3 and APACHE

II scores to assess the predictive accuracy of these models and to determine the in-hospital mortality for each type of transplant (19).

To assess discrimination (capacity to classify survivors and non-survivors), sensitivity and specificity were calculated for the different SAPS 3 and APACHE II scores by plotting the ROC curve and calculating the respective area under the curve.

The best discriminating value was determined by maximal sensitivity and specificity. The highest value that resulted from this product was used as the cutoff point, and 95% confidence intervals (95% CIs) were computed for true-and false-positive rates and for the correct classification rate of the outcomes.

The Hosmer-Lemeshow goodness-of-fit test was used to assess calibration (the agreement between the observed and expected numbers of survivors and non-survivors with regard to the probability of death) (20). In this analysis, p>0.05 was believed to indicate good test adjustment.

The standardized mortality ratio (SMR) was calculated by dividing the observed mortality rate by the predicted mortality rate. An SMR equal to 1.0 indicated that the number of observed deaths equaled that of the expected number of deaths; an SMR>1.0 indicated the occurrence of a greater number of deaths than expected. Calibration and discrimination were assessed (as previously described) for the original models.



The study included 501 postoperative transplant patients admitted to the ICU between May 2006 and January 2007.

The mean patient age was 46 ± 2 years old; extubation occurred within 24 hours of admission (institutional protocol); the median ICU stay was three days (25th, 50th, and 75th percentiles of two, three, and five days, respectively), and the median hospital stay was 21 days (25th, 50th, and 75th percentiles of 14, 21, and 35 days, respectively).

The sample included 152 (30%) liver transplant patients (116 male, 36 female), 271 (54.9%) kidney transplant patients (168 male, 103 female), 54 (10.7%) lung transplant patients (29 male, 25 female), and 24 (4.7%) kidney-pancreas transplant patients (15 male, 9 female).

The ranges of the APACHE II and SAPS 3 scores among survivors and non-survivors are shown in Table 1.

APACHE II significantly overestimated the mortality rates for all of the transplant patients. An analysis by type of transplant showed that the score underestimated the mortality rate in lung transplant patients, which was 52% greater than the predicted rate, with no statistical significance. In the other cases, the score overestimated mortality non-significantly, except for renal transplant patients (Table 2).

The SAPS 3 global equation underestimated the mortality rate in all types of transplants and showed statistical significance in cases of renal and pulmonary transplants (Table 2). The same poor performance was observed after customizing for South America, underestimating mortality in all of the groups and underestimating it statistically significantly in the same groups, as with the SAPS 3 global equation (Table 2).

For score calibration, the Hosmer-Lemeshow (HL-H and HL-C) tests were applied to all transplant groups; the test showed performance differences and overestimated the deciles for both scores (Table 3).

For score discrimination, an ROC curve analysis was used (Figure 1). Mortality discrimination was poor in the scores for both the liver and renal transplant groups and was average for the lung transplant group. The kidney-pancreas sample was too small for analysis.

ROC curve analysis indicated a cutoff point above which the discrimination to predict mortality increased significantly for each group. For the liver transplant patients, SAPS 3>40 had sensitivity of 0.6 and specificity of 0.46 in predicting mortality, whereas APACHE II>16.5 had sensitivity of 0.67 and specificity of 0.63. In the lung transplant group, SAPS 3>31.5 had sensitivity of 0.69 and specificity of 0.73 in predicting mortality, whereas the sensitivity and specificity of APACHE II$17.5 were 0.69 and 0.76, respectively.



In the present study, no differences were observed between SAPS 3 and APACHE II scores regarding their ability to predict hospital mortality in the transplant groups analyzed, even after customizing for South America. A study by Silva Junior et al. showed accuracy in predicting mortality with the South America-customized SAPS 3 equation in two surgical ICUs in Brazil (21). Their results did not match those of this study, most likely because transplant patients present a different surgical profile from that of general ICU patients; therefore, the two patients cohorts should be studied separately.

In terms of hospital mortality, the discrimination of both scores was poor for the liver and kidney transplant groups and average for the lung transplant group. Both score calibrations were poor for all of the groups.

These results are consistent with the literature, in which some studies have previously reported that SAPS models and APACHE scores are likely to overestimate mortality in high-risk patients and underestimate mortality in low-risk patients (2,8).

Liver transplant group

The mortality of our liver transplant patients was overestimated by APACHE II. Several studies that used the APACHE system in liver transplants have shown opposite results. Spanier et al. (23) used APACHE II to assess 102 patients with end-stage liver disease awaiting transplantation; the authors reported low accuracy of this score for predicting the risk of death. Bein et al. (24) and Sawyer et al. (25) reported a correlation between mortality and APACHE

II scores, but predicted mortality was not calculated in either study (25).

Angus et al. (26) used APACHE II in a group of liver transplant patients and concluded that the score was a good predictor of long-term (1-year) mortality after liver transplantation. An attempt to calibrate APACHE II for postoperative liver transplant patients (25-28) resulted in the overestimation of mortality, although to a lesser degree when compared with the original model. Arabi et al. (27) used the recalibration of APACHE II proposed by Angus et al. (26), and the results were the same as those obtained in the study by the latter group.

Keegan et al. (30) evaluated the performance of APACHE III in liver transplant recipients and demonstrated poor performance.

Recently, Park et al. (29) applied APACHE II and APACHE III scores in urgent liver transplant patients with acute liver failure. There was weak mortality prediction in this patient group, with mean scores (survivors 22.50 ± 5.89 and non-survivors 18.19 ± 5.89) and areas under the ROC curve (0.713 and 0.737 with p<0.05) that were similar to those found in our study.

No studies have reported results using SAPS 3 scores in liver transplant patients.

Kidney transplant group

Mortality was also overestimated by APACHE II in our kidney transplant patients. Past evidence has indicated that APACHE II was not adequate for evaluating mortality in this group (25,33).

No previous studies reporting the use of SAPS 3 in kidney transplant patients were available for a comparison with our findings.

We believe that some admission variables applied to SAPS 3 scores, such as creatinine, mechanic ventilation and the need for kidney repositioning, are not predictors of death (36); this belief may explain the overestimation of mortality in this group.

Lung transplant group

Slightly better performance was observed for APACHE II and SAPS 3 in lung transplant patients. However, these results cannot be compared with those of previous studies because there have been no reports in the literature regarding the application of APACHE II disease severity scores in this specific group. In validation studies focused on mortality scores, only SAPS 3 was used with this patient group (in small numbers) (11,12).

Some important advantages of this study included the short data collection period, the prevention of possible biases resulting from medical and technological improvements that occurred during the study period, and the fact that all of the transplant patients came from a single service that used standardized clinical and surgical methods and techniques.

The present results should be interpreted carefully. Our study had a number of limitations. The patients were cared for at a single center and thus might not be representative of other ICUs in other medical centers. While this fact reflects the referral practice of our institution, it might limit the generalization of our findings.

However, the validation of SAPS 3 in a group of patients at a single center might more accurately reflect the model performance, without the confounding influence of differing standards of care at different institutions.

In contrast, the low number of deaths limited the power of the calibration analysis.

We believe that the low accuracy of SAPS 3 and APACHE II in transplant patients stems from these systems underscoring physiologic aspects, such as creatinine level and liver function, which are usually (and by definition) poor in transplant patients before they receive new organs and can persist into the immediate postoperative period (28). However, because these variables tended to improve rapidly following transplantation, the scores overestimated mortality in the end.

Our results clearly demonstrate the need for a severity of illness classification system that specifically focuses on transplant patients, as well as on patient subgroups receiving different types of transplants.

Although this transplant sample was sufficiently large to assess illness severity scores, it was still too small for the design of a new predictive model. In addition, the results described here refer to one single ICU located in one specific country and region.

We recommend using different scores that consider each transplant's peculiar characteristics that contribute to posttransplantation mortality, such as MELD score variables for liver transplant patients (27-30), which account for creatinine, prothrombin time (INR) and bilirubin. For example, SAPS3 does not consider prothrombin time, a liver function marker that is an important variable. Studies that used MELD to predict post-liver transplant survival have shown conflicting results (31-35). These results could have been affected by using only pre-operative variables, thereby neglecting perioperative variables, such as surgical technique, ischemic time, graft conditions, and the need for immunoglobulin.

The Lung Allocation Score (LAS) has demonstrated great accuracy in predicting short-term mortality increases, survival decreases and increases in complications in the first year after lung transplants (34-36). Although this score was not developed for lung transplant patients, using some of its variables could influence the results in this patient group.

Therefore, a combination of variables from SAPS 3 (post-operatory) and LAS (pre-operatory) and other perioperative factors might help to define a better score for use in this specific subgroup.

Finally, to the authors' knowledge, this was the first study using SAPS 3 in transplant patients to demonstrate that this score did not predict hospital mortality in this population.

The present findings indicate that the accuracy of SAPS 3 is as low poor as that of APACHE II in predicting in-hospital mortality in postoperative transplant patients who are admitted to the ICU and overestimated patient mortality. The poor discrimination and calibration obtained with SAPS 3 and APACHE II do not warrant the use of these scoring systems in this patient group.

The lack of any similar studies precluded a comparison of our results with those of other researchers. Further studies will be necessary to validate our results.



Oliveira VM was responsible for the data acquisition, analyses and writing of the manuscript. Brauner JS was responsible for interpreting the data and writing the manuscript. Filho ER, Susin RG, Draghetti V, and Bolzan ST were responsible for data acquisition. Vieira SR was responsible for the data interpretation and writing the manuscript.



1. Costa JI, Gomes do Amaral JL, Munechika M, Juliano Y, Bezerra Filho JG. Severity and prognosis in intensive care: prospective application of the APACHE II index. Sao Paulo Med J. 1999;117(5):205-14.         [ Links ]

2. Diamond GA. What price perfection? Calibration and discrimination of clinical prediction models. J Clin Epidemiol. 1992;45(1):85-9,         [ Links ]

3. Glance LG, Osler T, Shinozaki T. Intensive care unit prognostic scoring systems to predict death: a cost-effectiveness analysis. Crit Care Med. 1998;26(11):1842-9,         [ Links ]

4. Higgins TL. Quantifying risk and benchmarking performance in the adult intensive care unit. J Intensive Care Med. 2007;22(3):141-56,         [ Links ]

5. Moreno RP. Outcome prediction in intensive care: why we need to reinvent the wheel. Curr Opin Crit Care. 2008;14(5):483-4,         [ Links ]

6. Pronovost PJ, Angus DC, Dorman T, Robinson KA, Dremsizov TT, Young TL. Physician staffing patterns and clinical outcomes in critically ill patients: a systematic review. JAMA. 2002;288(17):2151-62,         [ Links ]

7. Sachdeva RC. Statistical basis and clinical applications of severity of illness scoring system in the intensive care unit. Curr Opin Crit Care. 1999;5:180,         [ Links ]

8. Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med. 2000;19(4):453-73,<453::AID-SIM350>3.0.CO;2-5.         [ Links ]

9. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med. 1985;13(10):818-29,         [ Links ]

10. Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, et al. The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest. 1991;100(6):1619-36,         [ Links ]

11. MetnitzPG,Moreno RP,AlmeidaE,JordanB,BauerP,Campos RA,etal. SAPS 3--From evaluation of the patient to evaluation of the intensive care unit. Part 1: Objectives, methods and cohort description. Intensive Care Med. 2005;31(10):1336-44,         [ Links ]

12. Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3--From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med. 2005;31(10):1345-55,         [ Links ]

13. Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006; 34(5):1297-310,         [ Links ]

14. Bronsther O, Fung JJ, Izakis A, Van Thiel D, Starzl TE. Prioritization and organ distribution for liver transplantation. JAMA. 1994;271(2):140-3,         [ Links ]

15. Deschenes M, Villeneuve JP, Dagenais M, Fenyves D, Lapointe R, Pomier-Layrargues G, et al. Lack of relationship between preoperative measures of the severity of cirrhosis and short-term survival after liver transplantation. Liver Transpl Surg. 1997;3(5):532-7,         [ Links ]

16. Eckhoff DE, Pirsch JD, D'Alessandro AM, Knechtle SJ, Young CJ, Geffner SR, et al. Pretransplant status and patient survival following liver transplantation. Transplantation. 1995;60(9):920-5.         [ Links ]

17. Soares M, Salluh JL, Validation of the SAPS 3 admission prognostic model in patients with cancer in need of intensive care. Intensive Care Med. 2006;32:1839-44,         [ Links ]

18. Soares M, Silva UVA, Teles JMM, Silva E, Caruso P, Lobo SMA, et al. Validation of four prognostic scores in patients with cancer admitted to Brazilian intensive care units: results from a prospective multicenter study. Care Med. 2010;36:1188-95,         [ Links ]

19. Martinez EZ, Louzada-Neto F, Pereira BB. A curva ROC para testes diagnosticos. Cad Saude Colet (Rio J). 2003;11:7-31.         [ Links ]

20. Chang RW. Predicting deaths among intensive care unit patients. Crit Care Med. 1988;16(1):34-42,         [ Links ]

21. Silva Junior JM, Malbouisson LMS, Nuevo HL, Barbosa LGT, Marubayashi LY, Teixeira IC, et al. Applicability of the Simplified Acute Physiology Score(SAPS 3) in Brazilian Hospitals. Rev Bras Anestesiol. 2010;60(1):20-31,         [ Links ]

22. Capuzzo M, Moreno RP, Le Gall JR. Outcome prediction in critical care: the Simplified Acute Physiology Score models. Curr Opin Crit Care. 2008;14(5):485-90,         [ Links ]

23. Spanier TB, Klein RD, Nasraway SA, Rand WM, Rohrer RJ, Freeman RB, et al. Multiple organ failure after liver transplantation. Crit Care Med. 1995;23(3):466-73,         [ Links ]

24. Bein T, Frohlich D, Pomsl J, Forst H, Pratschke E. The predictive value of four scoring systems in liver transplant recipients. Intensive Care Med. 1995;21(1):32-7,         [ Links ]

25. Sawyer RG, Durbin CG, Rosenlof LK, Pruett TL. Comparison of APACHE II scoring in liver and kidney transplant recipients versus trauma and general surgical patients in a single intensive-care unit. Clin Transplant. 1995;9(5):401-5.         [ Links ]

26. Angus DC, Clermont G, Kramer DJ, Linde-Zwirble WT, Pinsky MR. Short-term and long-term outcome prediction with the Acute Physiology and Chronic Health Evaluation II system after orthotopic liver transplantation. Crit Care Med. 2000;28(1):150-6,         [ Links ]

27. Arabi Y, Abbasi A, Goraj R, Al-Abdulkareem A, Al Shimemeri A, Kalayoglu M, et al. External validation of a modified model of Acute Physiology and Chronic Health Evaluation (APACHE) II for orthotopic liver transplant patients. Crit Care. 2002;6(3):245-50,         [ Links ]

28. Moreno R. The customization of APACHE II for patients receiving orthotopic liver transplants. Crit Care. 2002;6(3):188-9,         [ Links ]

29. Park KM, Kim KH, Lee SG, Lee YJ, Hwang S, Ahn CS, et al. Clinical value of APACHE scores in patients who require intensive care before liver transplantation. Hepatogastroenterology. 2008;55(88):2135-9.         [ Links ]

30. Keegan MT, Gali B, Findlay JY, Heimbach JK, Plevak DJ, Afessa B. APACHE III outcome prediction in patients admitted to the intensive care unit after liver transplantation: a retrospective cohort study. BMC Surg. 2009;9:11,         [ Links ]

31. Onaca NN, Levy MF, Netto GJ, Thomas MJ, Sanchez EQ, Chinnakotla S, et al. Pretransplant MELD score as a predictor of outcome after liver transplantation for chronic hepatitis C. Am J Transplant. 2003;3(5):626-30,         [ Links ]

32. Jacob M, Copley LP. Lewsey JD, Gimson A, Toogood GJ, Rela M, et al. Pretransplant MELD score and post liver transplantation survival in the UK and Ireland. Liver Transplant. 2004;10(7):903-7,         [ Links ]

33. Hayashi PH, Forman L, Steinberg T, Wachs M, Kugelmas M, et al. Model for End-Stage Liver Disease score does not predict patient or graft survival in living donor liver transplant recipients. Liver Transplant 2003;9(7):737-40,         [ Links ]

34. Desai NM, Mange KC, Crawford MD, Abt PL, Frank AM, Markmann JW et al. Predicting outcome after liver transplantation: utility of the Model for End-Stage liver Disease and a newly derived discrimination function. Transplantation. 2004;77(1):99-106,         [ Links ]

35. Wong CS, Lee WC, Jenq CC, Tian YC, Chang MY, Lin CYL, et al. Scoring Short-Term Mortality after Liver Transplantation. Liver Transpl. 2010;16(2):138-46,         [ Links ]

36. Sadaghdar H, Chelluri L, Bowles SA, Shapiro R. Outcome of renal transplant recipients inthe ICU. Chest. 1995;107(5):1402-5,         [ Links ]

37. Weiss E.S, Allen JG, MCA, Conte JV, Shah AS. Lung allocation score predicts survival in lung transplantation patients with pulmonary fibrosis. Ann Thorac Surg. 2009;88(6):1757-64,         [ Links ]

38. Aranuotakis GJ, Allen JG, Merlo CA, Sullivan BE, Baumgartner WA, Conte JV e Shah AS. Impact of the lung allocation score on resource utilization after lung transplantation in the United States. J Heart Lung Transplant. 2011;30(1):14-21,         [ Links ]

39. Smits JM, Mertens BJ, Van Houwelingen HC, Haverich A, Persijn GG, Laufer G. Predictors of lung transplant survival in eurotransplant. Am J Transplant. 2003;3(11):1400-6.         [ Links ]

40. Takahashi SM, Garrity ER. The impact of the lung allocation score. Semin Respir Crit Care Med. 2010;31(2):108-14,         [ Links ]

41. Vermuelen KM, van der Bij W, Erasmus ME, TenVergert EM. Long-term health-related quality of life after lung transplantation: different predictors for different dimensions. J Heart Lung Transplant. 2007; 26(2):188-93,         [ Links ]



Received for publication on September 6, 2012
First review completed on September 19, 2012
Accepted for publication on October 15, 2012
No potential conflict of interest was reported.



Tel.: 55 51 3334-4201

Creative Commons License All the contents of this journal, except where otherwise noted, is licensed under a Creative Commons Attribution License