Survival analysis of elderly patients in Intensive Care Units

Abstract

Objective:

Conduct a survival analysis of elderly patients hospitalized in an intensive care unit (ICU), identifying the predictors of mortality among this age group.

Methods:

A retrospective cohort study was performed with data from the medical records of 457 elderly patients hospitalized in an ICU located in the city of Natal in Brazil. Survival functions were estimated using the Kaplan-Meier estimator, and the Log-rank test was used for comparisons. In addition, a multiple Cox proportional hazards model was constructed to identify the independent effects of the predictors of survival.

Results:

It was found that the survival of elderly ICU patients declined due to factors such as increased hospitalization time, advancing years, unmarried (including common-law-marriage) status, the presence of shock, pneumonia, septicaemia, fractures, a reduced state of consciousness, hospitalization for clinical reasons, being bedridden prior to hospitalization, fever, bradycardia, hypotension, cardiac arrest and the need for mechanical ventilation. The multiple Cox proportional hazards model revealed that variables such as shock, longevity, bradycardia, fractures, fever, hospitalization in the public healthcare system and admission for clinical reasons remained significant as predictors of reduced survival in intensive care units.

Conclusions:

The survival rates of elderly persons in an ICU in the city of Natal in Brazil were affected by demographic and clinical predictors, and those related to the type of hospitalization and the health care network. This shows that any initiative aimed at increasing the survival of elderly ICU patients must look at individual and social issues and factors related to the health care network.

Keywords:
Intensive Care; Health of the Elderly; Survival Analyses.

Resumo

Objetivo:

realizar uma análise de sobrevida de pacientes idosos internados em Unidades de Terapia Intensiva (UTI) adulto, destacando os fatores preditores de óbito nesse grupo etário.

Método:

estudo de coorte retrospectivo que coletou dados em 457 prontuários de idosos internados em UTI na cidade de Natal, RN, Brasil. As funções de sobrevida foram calculadas por meio do estimador de Kaplan-Meier, utilizando o teste de Log-rank para estabelecer comparações. Além disso, foi feito um modelo múltiplo de riscos proporcionais de Cox para a identificação do efeito independente dos preditores de sobrevida.

Resultados:

Foi verificado que o maior tempo de internação, idade avançada, não ter união estável, apresentar choque, pneumonia, sepse, fratura, rebaixamento de nível de consciência, internação por motivo clínico, estar acamados antes da internação, com febre, bradicardia, hipotensão, ter parada cardiorrespiratória e necessitar de ventilação mecânica diminuiu a sobrevida de idosos internados em terapia intensiva. O modelo múltiplo proporcional de Cox revelou que as variáveis choque, idoso longevo, com bradicardia, fratura, febre, internação na rede pública e admissão por motivo clínico permaneceram significativos como fatores prognósticos de menor sobrevida de idosos em terapia intensiva.

Conclusões:

A sobrevida dos idosos internados em UTI de Natal, RN é afetada por fatores prognósticos de origem demográfica, clínica, referentes ao tipo de internação e à rede de serviços de saúde. Isso demonstra que ações que visem ao aumento da sobrevida de idosos em terapia intensiva precisam discutir questões individuais, sociais e referentes à rede de atenção à saúde.

Palavras-chave:
Terapia Intensiva; Saúde do Idoso; Análise de Sobrevida.

INTRODUCTION

One of the biggest challenges for health in Brazil is the construction of a network of services that can directly influence the social determinants of the health of the population and reduce the inequalities that impact on ways of living and dying, especially among socially vulnerable population groups such as the elderly.

This assertion is supported by the fact that the elderly in both Brazil and the rest of the world, especially those from low income groups, exhibit the poorest self-evaluations of health, high levels of functional disability and have difficulty accessing consultations, exams and hospitalizations¹. An aggravating factor in Brazil is that the proposed universal public access offered by the Sistema Único de Saúde (the Unified Health System) (SUS) is still in the process of effectuation, as spending on private health care is greater than government investment in the sector, which affects both family income and, often, the efficacy of the services provided^2,3.

SUS underfunding also increases the harm done to the health of the elderly. It is the responsibility of the Ministry of Health, as a managerial body, to plan and implement improvements in the short, medium and long term, always considering the need for greater investment, faced with a scenario of population aging and a higher prevalence of chronic diseases that cause frailty and death⁴. In this context, Brazil also needs to invest in the construction of intersectoral public policies that guarantee the promotion, prevention, treatment and rehabilitation of health, taking into account the needs of different age groups, with a view to tackling the social determinants inherent in the health/disease process of each such group⁵.

The hospitalization of the elderly in an Intensive Care Unit (ICU) is one facet of this reality, as such units have become increasingly expensive in the context of a hospital-centered health care model, restricting their capacity for innovation to technological areas and neglecting aspects such as the links with the network of preventive services, for example³. The role of intensive care in the health care of the elderly has expanded precisely because of the lack of preventive services. As an example of this, 52% of ICU hospitalizations in Brazil involve elderly persons, who in turn represent 60% of the overnight stays and financial resources available in adult ICUs, with a mortality rate that climbs as high as 62%, while the number of deaths among adult individuals in the sector is about 25%^6,7.

The causes of the disparity in these mortality figures is not exclusively clinical, as health professionals also face difficulties in establishing precise criteria for the admission of the elderly to ICUs. These doubts stem from educational, ethical, legal, religious, cultural and social dilemmas that are not systematically dealt with during the training of health professionals. Most of the time, during academic training, students in the field focus on technical discussions about ways to prolong life, where bioethical questions about terminality and finitude are superficially considered in the point-by-point and fragmented approach of the course curriculum⁸.

Prognostic scores could therefore be an auxiliary tool in decision-making about the hospitalization of elderly patients in the ICU. However, it is important that the application of such measures is accompanied by good clinical practice, the support of complementary exams, the health history of elderly individuals, a sense of ethics and the capacity to manage the resources available in the hospital institution to support each individual case in its singularity, avoiding negligent actions such as unnecessary hospitalizations and the misuse of resources.

The ICU is a space for seriously ill patients with a chance of recovery and not for the palliative care of terminally and irreversibly sick patients⁸. Thus, this article aims to carry out a survival analysis of elderly patients hospitalized in adult ICUs, highlighting the factors that impact the heightened mortality rates among this age group. The analysis of the object of study is relevant as it applies directly to the discussion regarding the recommendation of hospitalization in ICUs and the optimization of the use of resources available in public and private health networks.

METHODS

A retrospective cohort study was carried out, with the purpose of identifying predictive factors of death among the elderly population admitted to ICUs. The study was carried out in the city of Natal in the state of Rio Grande do Norte. Data collection was based on the medical records of elderly patients hospitalized between November 1, 2013 and January 31, 2014 in the ICUs of the participating institutions.

The data were collected from 155 ICU beds, of which 101 were from the SUS and 54 were from private institutions. This total represented 62% of the ICU beds available to the population of Natal, Rio Grande do Norte during the field phase of the study. Data were collected between January and August 2014, and the data of all subjects who complied with the inclusion and exclusion criteria were collected in the survey, totaling 457 hospitalizations of elderly persons. To be included in the study, subjects were required to be aged 60 years or older in the period of hospitalization and to have been admitted to the ICU within the temporal cut-offs of the study. Medical records that reported discharge or death in the ICU before 24 hours of hospitalization was completed were excluded, as in this case a Hospitalization Authorization (HA) was not generated in the medical records of the SUS ICUs, meaning that such occurrences were not classified as hospitalizations from a legal perspective in this sector. Outliers of the length of hospital stay variable were also excluded to improve the accuracy of the multiple analysis.

Prior to the data collection period, a pilot study was carried out with the purpose of adjusting the collection instrument, assessing the plausibility of the set of variables listed and delimiting the temporal cut-off of the study⁹. It should be stated that the three-month period for data collection established in the pilot study did not result in seasonal bias, as the prevalence of the hospitalization of elderly patients in ICUs is high and its incidence rate is not proven to be seasonally related¹⁰.

The dependent variable was divided into two categories, death and non-death during the follow-up period. The time until the event was counted in complete days from ICU admission (number of days recorded on the HA), starting with the hospitalization of the elderly in the ICU and with monitoring continuing for a period of 20 days. This period of time is justified by the fact that international studies of survival usually consider two groups of predictors of mortality: impacting factors that have an effect in up to 30 days, known as short-terms; and long-terms, which are variables that impact survival after 30 days from the date of admission. It was also observed in other studies that the average length of hospitalization of the elderly in ICUs, including standard deviations, is around 20 days^11-13. Remaining faithful to these assumptions is important for discussion and comparison between different studies.

The independent variables analyzed were gender, age, self-declared ethnicity/skin color, occupation, religion, place of residence, main diagnoses or diagnostic hypotheses, previously diagnosed chronic diseases which were not the reason for the current hospitalization, signs and symptoms present during ICU admission, length of hospital stay, and type of ICU bed (public or private).

Therefore, a descriptive analysis of the dependent variables in relation to the time of hospitalization was carried out. The survival functions were calculated using the Kaplan-Meier estimator, with and without stratification, using the Log-rank Test to compare the functions at 3, 6 and 9-day intervals. The descriptive analysis showed that the outcomes of 78% of admissions occurred in up to nine days. This justifies the choice of the upper limit of the cited time interval. The lower limit of three days is based on the definition of Hospital Infection, characterized as infections that manifest themselves clinically within three days after hospitalization or the discharge of the patient from the health institution¹¹. This was taken into account as both the infection and sepsis variables were found to be important during preliminary analyzes of the database.

The prognostic factors were calculated using hazard ratios (HR), taking into account 95% confidence intervals and the Cox proportional hazards model. Only variables with a crude HR with p<0.20 were included in the multiple analysis.

After the application of the Wald tests and the likelihood ratio test between the adjusted models, the Cox proportional hazards (hazard ratio) model was used to identify the independent effect of the variables which explained survival. The assumption of proportionality of the risks was assessed by Schoenfeld graphical analysis. The relevance of each variable included in the multiple model was verified by Wald statistics, with variables with statistical significance remaining in the final model (p<0,05).

The initial project was sent to the Ethics Research Committee of the Hospital Universitário Onofre Lopes (CEP-HUOL) before the field work, in accordance with the directives of Resolution 466/12, which guides the carrying out of studies involving humans in Brazil. The research project was approved by the CEP-HUOL under CAAE N° 20578913.1.0000.5292. The FICF waiver was requested from the CEP-HUOL, while the signature of documents authorizing the access of the researchers to patient record data was requested from the institutions, with the anonymity of the names of the subjects enrolled in the research guaranteed.

RESULTS

A total of 457 elderly patients were admitted to the ICU, with a mean age of 74.23 (±9.78), of whom 108 (23.6%) died. The average length of stay in the ICU was 5.03 (±4.78) days. When stratified by discharge and death, however, means of 7.60 (±5.64) and 4.60 (mean±4.18) days, respectively, were observed. The variables occupancy, religion and place of residence were missing in more than 90% of cases due to incomplete medical records, and were not considered during the analysis.

At the end of the follow-up period, the overall survival rate (Figure 1) without stratification was 90.99% after three days, 80.01% after six days and 66.64% after nine days of hospitalization. A gradual decline of the accumulated survival rate was observed over the same period, when comparing the functions in relation to the sociodemographic variables related to hospitalization (Table 1). A lower survival rate was observed among elderly persons in the age group of 80 years or older, who were unmarried (including civil marriages), and who were hospitalized in the SUS.

Figure 1
Overall survival rate of elderly persons in intensive care units. Natal, Rio Grande do Norte, 2015.

Table 1
Survival table of elderly persons hospitalized in ICUs based on sociodemographic variables and variables related to hospitalization. Natal, Rio Grande do Norte, 2015.

Variables	Deaths (%)	3 days	6 days	9 days	p value*
Gender
Male	48 (44.44)	93.42	80.72	64.42	0.968
Female	60 (55.56)	88.76	79.34	68.57
Age (years)
60 to 80	59 (54.63)	92.94	82.85	68.25	0.002
81 or more	49 (45.37)	86.05	73.33	62.72
Marital status
Married (including common-law marriage)	43 (39.81)	94.05	82.64	71.92	<0.001
Unmarried (including common-law marriage)	55 (50.93)	85.96	69.71	50.92
Network
Private	53 (49.07)	92.28	84.41	73.53	0.015
SUS	55 (50.93)	88.68	73.56	57.04

• * Student's T-test

In relation to clinical and morbidity variables, elderly persons who were diagnosed with pneumonia or shock, were suffering from sepsis, were hospitalized for a fracture, were disoriented, hospitalized due to a clinical condition, had bradycardia, were bedridden before hospitalization, used artificial ventilation or suffered cardiorespiratory arrest, fever or hypotension had a lower survival rate (Table 2).

Table 2
Survival table of elderly persons hospitalized in ICUs based on clinical and morbidity variables. Natal, Rio Grande do, 2015.

Variables	Deaths (%)	3 days	6 days	9 days	p value*
Pneumonia
No	72 (66.67)	92.98	83.67	68.65	0.012
Yes	36 (33.33)	80.01	63.79	55.99
Shock
No	84 (77.78)	92.93	84.53	70.11	<0.001
Yes	24 (22.22)	66.67	39.89	34.90
Sepsis
No	68 (62.96)	93.33	88.08	74.05	<0.001
Yes	40 (37.04)	76.27	46.77	37.20
Fractures
No	98 (90.74)	92.18	81.26	67.18	0.020
Yes	10 (9.26)	67.99	54.39	54.39
Bedridden
No	77 (71.30)	92.03	81.37	69.59	0.003
Yes	31 (28.70)	85.75	73.50	55.96
Type of hospitalization
Surgical	22(20.37)	95.62	85.54	77.15	0.032
Clinical	86 (79.63)	88.76	77.63	62.66
Level of consciousness
Orientated	30 (27.78)	96.33	90.44	76.67	<0.001
Disoriented	23 (21.30)	94.05	85.92	74.64
Unconscious	55 (50.93)	78.79	61.67	48.80
Bradycardia
No	91 (84.26)	92.01	80.13	70.28	0.017
Yes	17 (15.74)	82.61	78.26	41.09
Hypotension
No	75 (69.44)	93.24	83.84	73.80	<0.001
Yes	33 (30.56)	80.83	65.50	43.71
Artificial Ventilation
No	47 (43.52)	97.23	91.95	78.79	<0.001
Yes	61 (56.48)	75.11	56.33	44.48
Cardiorespiratory arrest
No	90 (83.33)	93.66	82.80	68.70	0.002
Yes	18 (16.67)	63.31	43.83	38.35
Fever
No	91 (84.26)	91.31	80.51	68.22	0.184
Yes	17(15.74)	88.93	76.23	55.74

• * Chi-squared test.

When Cox regression was performed, presenting simple and adjusted proportional hazard values (Table 3), the elderly persons who remained in the model and had a higher risk of death in the ICU were those who exhibited shock (140%), were aged 80 or older (95%), suffered from bradycardia (131%), were hospitalized in SUS beds (114%), had fractures (108%), were hospitalized for clinical reasons (102%) or had fever (91%).

Table 3
Multiple proportionate hazard model for deaths in ICU, with simple and adjusted hazard ratios for the study variables, Natal, Rio Grande do Norte, 2015.

Variables	HRa	HR Ajb	IC 95%c	p Value 4
Shock
No	1.00
Yes	2.87	2.40	1.49 - 3.88	<0.001
Age (years)
60 to 79	1.00
80 or more	1.76	1.75	1.17 - 2.64	0.007
Bradycardia
No	1.00
Yes	1.84	2.31	1.33 - 4.00	0.003
Network
Private	1.00
SUS	1.57	2.14	1.42 - 3.21	<0.001
Fractures
No	1.00
Yes	2.07	2.08	1.02 - 4.24	0.043
Type of hospitalization
Surgical	1.00
Clinical	1.64	2.02	1.21 - 3.37	0.007
Fever
No	1.00
Yes	1.40	1.91	1.11 - 3.27	0.019

• a: Hazard ratios; b: Adjusted hazard ratios; c: Confidence interval of 95%; d: Wald Test.

The variables infection, acute respiratory insufficiency, acute myocardial infarction, dyspnea, cyanosis, edema and comorbidities not directly related to the current hospitalization were not considered predictors for the death of the elderly persons in the ICU in the present study.

DISCUSSION

Elderly persons hospitalized in the ICUs for longer periods, who were older, unmarried (including common-law marriages), hospitalized in the SUS, diagnosed with shock, pneumonia, sepsis, fractures, had a reduced level of consciousness, were affected by clinical disorders, bedridden before hospitalization, exhibited fever, bradycardia, hypotension, cardiorespiratory arrest and required mechanical ventilation had lower survival rates during the study period.

However, when these findings were subjected to the Cox proportional multiple model, it was observed that only the variables shock, older elderly persons, bradycardia, fractures, fever, hospitalization via the public network and admission due to clinical reasons remained as prognostic factors for lower survival rates among elderly persons admitted to the ICU. Comorbidities and previously diagnosed chronic diseases unrelated to the reason for the current hospitalization were not associated with the survival of the elderly persons at any of the analysis levels.

Acute clinical disorders and factors such as the longevity of patients appear to be more significant in survival studies with a follow-up period of less than 30 days. In this range, conditions such as level of consciousness, respiratory diseases and mechanical ventilation were identified as predictors of mortality. In studies with a follow-up period of more than 30 days, comorbidities are associated with a worse survival prognosis, which is consistent with the results presented here. However, the variable sepsis is equally significant in survival studies with both short and long follow-up periods, including those of up to five years, which highlights the impact of this variable on the mortality of elderly patients in ICUs^12-14.

However, the present study found that the chronic condition of being bedridden at home before hospitalization was significant for the survival of elderly persons in the ICU, even over a period of less than 30 days. This demonstrates that the impairment of functional capacity is an important element for the quality of life of the elderly and is associated with a greater probability of death during the hospitalization of individuals of this age group¹⁴.

At the same time, the association of the functional decline of the elderly with falls and fractures is a commonly discussed theme in studies in geriatrics and gerontology^15,16. In addition, death due to post-fracture hospitalization has attracted the attention of researchers, especially as such events have intensified in age groups younger than 75 years^17,18. In this scenario, where fractures also appear as a prognostic factor for the death of elderly persons in the ICU, it is important to consider strategies for the prevention of traumatic events, even for elderly individuals who are considered robust.

In some situations, surgical intervention is necessary in the case of fractures among the elderly, and the intensive therapy unit is the site of immediate post-surgical recovery for the most severe cases, which in the present study were considered as hospitalizations for surgical reasons. However, hospitalizations for clinical reasons appeared as a risk in the survival table and in the multiple Cox model, a finding also obtained by other studies^12,14,18. It is therefore assumed that the elective and controlled nature of non-emergency surgical procedures guarantee lower mortality than clinical events that usually affect the elderly in an acute and unexpected way.

Regarding shock and cardiorespiratory arrest, the study by Fuchs et al.¹⁸ presented results similar to those of the present work. Such variables, clinically and physiologically related to bradycardia and hypotension, which were also associated with a decrease in the survival of the elderly in the ICU, are prognostic factors of death, even after a one year follow-up period¹⁸.

SUS patients had a lower survival rate than elderly persons hospitalized in the private system. To understand this, it is necessary to go beyond hospitals and ICUs, and to perceive these institutions in the context of public health policies in Brazil. The public health system faces financial problems that directly affect the quality of services provided, including health care for the elderly. However, addressing this crisis also involves an approach that goes beyond economic factors, such as: breaking away from the fragmented care model, seeking interdisciplinary care, improving resource management and professional training systems¹⁹.

CONCLUSIONS

A retrospective study based on medical records presents a potential limitation in relation to possible calibration bias. However, this study considered final diagnoses, usually identified following examination and analysis by various health professionals from a range of areas. In addition, the rigor of the Federal Medical Council regarding the correct completion of death certificates has improved death information systems in Brazil.

The survival of elderly persons hospitalized in ICUs in Natal, Rio Grande do Norte is affected by different prognostic factors represented by demographic and clinical variables related to the type of hospitalization and to the network of health services. The diversity of the classification of these variables indicates the complexity of the phenomenon studied. It also suggests that any action aimed at increasing the survival of elderly persons in ICUs needs to go beyond the clinical approach and function both on an individual level and within the social context of individuals and in the health care network.

However, there is a need for further studies to contribute to this discussion. The importance of performing cohort studies with elderly patients hospitalized in ICUs with a longer follow-up period, including the post-discharge period, is important. In this way, it would be possible to perceive not only death/discharge rates, but also the quality and the years of life added after the treatment offered to the elderly.

REFERENCES

• ¹
  Lima e Costa MFF, Facchini LA, Matos DL, Macinko J. Mudanças em dez anos das desigualdades sociais em saúde dos idosos brasileiros (1998-2008). Rev Saúde Pública. 2012;46 (Supl):100-7.
• ²
  Boing AC, Bertoldi AD, Posenato LC, Perez KG. Influência dos gastos em saúde no empobrecimento de domicílios no Brasil. Rev Saúde Pública. 2014;48(5):797-807.
• ³
  Barbosa PR, Gadelha CAG. O papel dos hospitais na dinâmica de inovação em saúde. Rev Saúde Pública. 2012;46(Supl):68-75.
• ⁴
  Mendes AN. Tempos turbulentos na saúde pública brasileira: impasses do financiamento no capitalismo financeiro. São Paulo: Hucitec; 2012.
• ⁵
  Fundação Oswaldo Cruz. A saúde no Brasil em 2030: prospecção estratégica do sistema de saúde brasileiro. Rio de Janeiro: Fiocruz; 2013.
• ⁶
  Pedrosa IL. Construção de um instrumento de avaliação prognóstica para idosos em unidade de terapia intensiva [tese]. Porto Alegre: Pontifícia Universidade Católica do Rio Grande do Sul; 2014.
• ⁷
  Oliveira ABF, Dias OM, Mello MM, Araújo S, Dragosavac D, Nucci A, et al. Fatores associados à maior mortalidade e tempo de internação prolongado em uma unidade de terapia intensiva de adultos. Rev Bras Ter Intensiva. 2010;22(3):250-6.
• ⁸
  Poletto S, Santina JS, Bettinellia LA. Dilemas do enfrentamento da morte de pacientes Idosos. Rev Ciênc Juríd Empres. 2012;13(2):49-55.
• ⁹
  Canhota C. Investigação passo a passo: perguntas e respostas para investigação clínica. Lisboa: APMCG; 2008.
• ¹⁰
  Brasil. Ministério da Saúde. Diretrizes metodológicas: análise de impacto orçamentário: manual para o Sistema de Saúde do Brasil. Brasília, DF: Ministério da Saúde; 2012.
• ¹¹
  Garcia LM, César ICO, Braga CA, Souza GAD, Mota EC. Perfil epidemiológico das infecções hospitalares por bactérias multidrogarresistentes em um hospital do norte de Minas Gerais. Rev Epidemiol Control Infect. 2013;3(2):45-9.
• ¹²
  Garland A, Olafson K, Ramsey CD, Yogendran M, Fransoo R. Distinct determinants of long-term and short-term survival in critical illness. Intensive Care Med. 2014;(40):1097-1105.
• ¹³
  Davis JS, He V, Anstey NM, Condon JR. Long term outcomes following hospital admission for sepsis using relative survival analysis: a prospective cohort study of 1,092 patients with 5 year follow up. PLos ONE. 2014;9(12):12-5.
• Taccone FS, Artigas AA, Sprung CL, Moreno R, Sakr Y, Vincent JL. Characteristics and outcomes of cancer patients in European ICUs. Crit Care. 2010;(13 (1):1-10.
• ¹⁵
  Soares DS. Fraturas de fêmur em idosos no Brasil: análise espaço-temporal de 2008 a 2012. Cad Saúde Pública. 2014;30(12):2669-78.
• ¹⁶
  Falsarella GR, Gasparotto LPR, Coimbra AMV. Quedas: conceitos, frequências e aplicações à assistência ao idoso. Rev Bras Geriatr Gerontol. 2014;17(4):897-910.
• ¹⁷
  Kannegaard PN, Mark SVD, Eiken P, Abrahamsen B. Excess mortality in men compared with women following a hip fracture. National analysis of comedications, comorbidity and survival. Age Ageing. 2010;(39):203-9.
• ¹⁸
  Fuchs L, Chronaki CE, Park S, Novack V, Baumfeld Y, Scott D, et al. ICU admission characteristics and mortality rates among elderly and very elderly patients. Intensive Care Med. 2012; 38(10):1654-61.
• ¹⁹
  Mendes J, Bittar O. Com 25 anos de duração, o Sistema Único de Saúde - SUS - enfrenta problemas no financiamento e na qualidade dos serviços produzidos. Seus desafios envolvem mudanças nas necessidades de saúde (transições demográfica e epidemiológica) e na tecnologia médica. Rev Fac Ciênc Méd Sorocaba [Internet]. 2012 [acesso em 20 Mar 2016];38(1):1654-61. Disponível em: http://revistas.pucsp.br/index.php/RFCMS/article/view/18597
  » http://revistas.pucsp.br/index.php/RFCMS/article/view/18597

Publication Dates

• Publication in this collection
  Mar-Apr 2017

History

• Received
  11 July 2016
• Reviewed
  29 Jan 2017
• Accepted
  11 Mar 2017