Cultural adaptation and reliability for Brazil of the Automated Telephone Disease Management: Preliminary results

Abstracts

OBJETIVOS: Traduzir, adaptar culturalmente para o Brasil o ATDM Satisfaction Scales e avaliar a confiabilidade da versão adaptada em adultos brasileiros com DM. MÉTODOS: Estudo metodológico, cujo processo de adaptação cultural incluiu: tradução, comitê de juízes, retrotradução, análise semântica e pré-teste. Este estudo incluiu uma amostra de 39 adultos brasileiros com DM cadastrados em um programa educativo do interior paulista. RESULTADOS: A versão adaptada do instrumento mostrou boa aceitação com fácil compreensão dos itens pelos participantes, com confiabilidade variando entre 0,30 e 0,43. CONCLUSÃO: Após a análise das propriedades psicométricas e finalização do processo de validação no País, o instrumento poderá ser utilizado por pesquisadores brasileiros, possibilitando ser comparado com outras culturas.

Telemedicina; Satisfação do paciente; Diabetes mellitus; Telefone

ORIGINAL ARTICLE

Cultural adaptation and reliability for Brazil of the Automated Telephone Disease Management: Preliminary results^*

Adaptación cultural y confiabilidad para el Brasil del Automated Telephone Disease Management: resultados preliminares

Talita Balaminut^I; Camila Aparecida Pinheiro Landim^II; Tânia Alves Canata Becker^II; Ellen Cristina Barbosa dos Santos^II; Gabriela Marsola Olivatto^III; Maria Lúcia Zanetti^IV; Carla Regina de Souza Teixeira^V

^INurse. Resident in Nursing in Neonatology, Department of Nursing Health Sciences Center, State University of Londrina - UEL - Londrina (PR), Brazil

^IINurse. Master's of Science. Doctoral Student in Graduate Nursing Fundamentals, Department of General and Specialized Nursing, Nursing School of Ribeirão Preto, University of São Paulo - USP - Ribeirão Preto (SP), Brazil

^IIIStudent, third-year of a Bachelor's Degree in Nursing, Nursing School of Ribeirão Preto, University of São Paulo - USP - Ribeirão Preto (SP), Brazil

^IVNurse. Free Professor (Livre-Docente). Associate Professor, Graduate Program in Nursing Fundamentals , Department of General and Specialized Nursing, Nursing School of Ribeirão Preto, University of São Paulo - USP - Ribeirão Preto (SP), Brazil

^VNurse. Post-Doctorate in Social Medicine and Advisor of the study in question. Associate Professor, Graduate Program in Nursing Fundamentals , Department of General and Specialized Nursing, Nursing School of Ribeirão Preto, University of São Paulo - USP - Ribeirão Preto (SP), Brazil

Corresponding Author

ABSTRACT

OBJECTIVES: To translate, culturally adapt for Brazil the Automated Telephone Disease Management (ATDM) Satisfaction Scales and evaluate the reliability of the adapted version in Brazilian adults with diabetes mellitus (DM).

METHODS: A methodological study whose cultural adaptation process included: translation, expert committee, back translation, semantic analysis and pretesting. This study included a sample of 39 Brazilian adults with DM enrolled in an educational program in São Paulo.

RESULTS: The adapted version of the instrument showed good acceptance with easy comprehension of the items by the participants, with reliability ranging between 0.30 and 0.43.

CONCLUSION: After analyzing the psychometric properties and finalizing the validation process in the country, the instrument can be used by Brazilian researchers, making it possible to compare with other cultures.

Keywords: Telemedicine; Patient satisfaction; Diabetes mellitus; Telephone

RESUMEN

OBJETIVOS: Traducir, adaptar culturalmente para el Brasil el ATDM Satisfaction Scales y evaluar la confiabilidad de la versión adaptada en adultos brasileros con DM.

MÉTODOS: Estudio metodológico, cuyo proceso de adaptación cultural incluyó: traducción, comité de jueces, retrotraducción, análisis semántica y pre-test.Este estudio incluyó una muestra de 39 adultos brasileros con DM registrados en un programa educativo del interior paulista.

RESULTADOS: La versión adaptada del instrumento mostró buena aceptación con fácil comprensión de los items por los participantes, con confiabilidad variando entre 0,30 y 0,43.

CONCLUSIÓN: Después del análisis de las propiedades psicométricas y finalización del proceso de validación en el País, el instrumento podrá ser utilizado por investigadores brasileros, posibilitando su comparación con otras culturas.

Descriptores: Telemedicina; Satisfacción del paciente; Diabetes mellitus; Teléfono

INTRODUCTION

The importance of chronic noncommunicable diseases (NCDs) in the current profile of population health is extremely important. Estimates from the World Health Organization (WHO) indicate that NCDs now account for 58.5% of all deaths and 45.9% of the total global disease burden expressed as lost years of healthy life ⁽¹⁾. Among these, diabetes mellitus (DM) is recognized as a global epidemic today, representing a major challenge for health systems around the world⁽²⁾.

Research conducted by the Diabetes Control and Complications Trial - DCCT (1993) and United Kingdom Prospective Diabetes Study Group - UKPDS (1998), showed that for both diabetes mellitus type 1 (DM1) and diabetes mellitus type 2 (DM2), the metabolic control with intensive treatment within certain limits can significantly reduce the development of complications ^(3,4).

To address the complexity of treatment, it is necessary to use innovative technologies in DM, since these provide better clinical outcomes. In this sense, the use of the telephone is an important strategy in health communication, with an expected increased application of this technology next year. The telenurse is a strategy of action in health that is different from nursing, representing a a leap forward from traditional nursing care

The Ministry of Health launched, in 2006, a manual with specific recommendations for integral care of diabetic patients and for the professional health team, recommending the following conduct: telephone contact between appointments, and planning services for emergency care of acute decompensation of glucose by telephone contact.

The international literature shows an increased effort in the insertion of new technologies for the care of person with DM. Studies with community participation in planning and evaluation of care are relevant, because they allow the feedback of interventions performed by health teams, improving health services. ⁽¹⁰⁾ Therefore, it is necessary to evaluate the developed work with new techonologies from the perspective of the person with DM.

Thus, the strategy of using the telephone in assisting people with NCDs seems to be a possibility to advance and monitor throughout the course of treatment, ensuring continuity of health and longitudinality of the care. Thus, there is a need to provide tools that enable the evaluation of health care services by telephone, or interventions such as educational programs for people with diabetes mellitus.

In this way, after literature review in national and international databases, it was evident that among the tools for evaluation in health care for chronic conditions using telephone was the Automated Telephone Disease Management (ATDM) Satisfaction Scales, whose aim was specifically to measure satisfaction of people with DM after submitting to intervention or educational programs by telephone.

The ATDM Satisfaction Scales instrument , originally developed in English by Dr. John Piette, consists of 11 items covering three areas: facility of completing the call (4 items), perceived usefulness of the call (3 items) and intrusiveness of the call (4 items). For each item of the instrument, five possible answers are offered, with scoring on a range of 1 to 5 points. The ranking of the instrument items occurs by variation of the Likert scale: Always (1); Mostly (2); Sometimes (3); Rarely (4) Never (5) ⁽¹¹⁾.

Considering the lack of instruments in order to assess this dimension of care in Brazilian culture, the present study aimed to translate and culturally adapt the ATDM Satisfaction Scales for Brazil and to present preliminary results on the reliability of the Brazilian version adapted for people with DM, despite the small convenience sample represented by the study.

METHODS

This study is characterized as a methodological investigation that included the search for new meanings, interpretations of phenomena, and development of data collection instruments ⁽¹²⁾ and to understand the cultural adaptation and reliability of an instrument that assessed the satisfaction of Brazilians with DM, after intervention or educational programs by telephone.

Although research on hypothetical and actual methodological investigations has shown that a minimum sample size of 50 subjects is sufficient to adequately represent and analyze the initial psychometric properties of an instrument to be tested in another country ⁽¹³⁾, the sample of our study included: 39 adult Brazilians, both genders, aged between 36 and 79 years, with 7.7% and 2.3% with DM2 and DM1, respectively, who were registered during the period of 2010-2011 in the Diabetes Education Program in the Nursing Education Center for Adults and Seniors, located in the Campus of the University of São Paulo in Ribeirão Preto-SP, and And linked to the Nursing School of Ribeirão Preto, University of São Paulo (EERP-USP). Subjects were selected using the following criteria: frequency greater than or equal to 75% participation in the educational groups of the Education Program on Diabetes, age greater than 18 years old, and agreed to participate.

A convenience sample was used, since participants were invited to participate in the study by phone, in the order in which they were enrolled in the Diabetes Education Program at the study site. Despite not being probabilistic, we were careful to maintain homogeneity between gender and age. Samples of this type can be considered representative of the population assisted in the service considered ⁽¹⁴⁾.

In regard to ethical issues, the present study was approved by the Ethics Committee for Human Research of the EERP, as Research Protocol No. 1.175/2010, with approval on August 20, 2010. Data were collected from March to May of 2011. In all interviews by telephone, the Terms of Free and Informed Consent (TFIC) were read and verbal consent requested of all participants, ensuring their privacy and the condition of strict confidentiality of their names.

To collect data, we proceeded in the following manner: initially, people were invited to participate in the study by telephone, and informed about the aims and purposes of the research by reading the TFIC. After verbal consent, the person was interviewed.

Thus, the directed interview was conducted by telephone and the ATDM Satisfaction Scales, translated into Brazilian Portuguese and adapted to the culture of the country, were used. Each interview had a mean time of 10 minutes, and the data were recorded using PacTel, a program that records telephone conversations. After interviews, the responses were manually typed and inserted in the instrument and database with validation and double entry, prepared in Excel of Microsoft Office 2010.

Following this, the medical records of the 39 participants were consulted for demographic data: age in years, date of birth, gender, marital status, occupation, education level, and family income.

Then the adaptation process adopted for the ATDM Satisfaction Scales instrument was performed according to the following steps: translation of the instrument into Brazilian Portuguese; obtaining the first consensus on the Portuguese version of the translations; evaluation by a committee of experts; backtranslation; obtaining the consensus of versions in English, and compared with the original one; semantic analysis of items, and a pre-test ⁽¹⁵⁾.

The translation of the ATDM Satisfaction Scales instrument in its original version (ATDM-OV) of English into Brazilian Portuguese was conducted individually by two bilingual persons with knowledge about the subject. The two versions were analyzed by the researchers involved in this study, resulting in the Consensual Portuguese Version 1 (ATDM-CPV1).

Thus, the evaluation was performed by an expert committee composed of researchers, health professionals, teachers with experience in the subject of care in diabetes mellitus and nursing communication, language professionals and translators, in order to assess the cultural, conceptual, semantic and idiomatic equivalence between the ATDM CPV1-and ATDM-OV instrument. The potential modifications of the equivalences were accepted, when they obtained a consensus of at least 85% approval by consensus of the total number of members of the expert committee, resulting in the Portuguese Consensual Version 2 (ATDM-PCV2).

The ATDM-PCV2 was then subjected to two individual translators, unaware of the study objectives, born in the United States of America and residents in Brazil, for the back translation. Then, the two back translated versions were analyzed in a meeting with both translators and researchers involved in the study and the final English version of the ATDM Satisfaction Scales instrument (ATDM-FEV) was determined, which was compared to ATDM-OV and submitted for assessment by the main instrument's author, Dr. John Piette. The comparison between ATDM-OV and ATDM-FEV versions resulted in no change in Portuguese Consensual Version 2 (ATDM-PCV2).

For the semantic analysis of the instrument items, the ATDM-PCV2 was implemented with five people with diabetes enrolled in the Diabetes Education Program of the EERP-USP. Then, the five selected people were invited to participate in this step, in order to properly understand the wording of the 11 items and the range of responses for the population for which the instrument was intended, in order to analyze the possible changes, additions and suggestions of the participants; no alterations were suggested.

Thus, it the ATDM-PCV2 was used in the pretest with five people with diabetes enrolled in the Diabetes Education Program at the study site. There was no need for modifications in completing or comprehension. Therefore, the process of cultural adaptation of the ATDM Satisfaction Scales instrument for Brazil was considered completed, maintaining the ATDM-PCV2 as the Final Portuguese Version (ATDM-FPV).

Finally, after completing the process of cultural adaptation and data collection, the reliability of ATDM-FPV was verified by the internal consistency of the items, calculated by Cronbach's alpha. It is the indicator most often used in the analysis of the internal consistency of instruments because it reflects the degree of covariance among the items themselves ⁽¹⁶⁾. The level of significance was set at 5% (α = 0.05).

RESULTS

The ATDM Satisfaction Scales instrument was translated and adapted into Brazilian Portuguese, to be employed in the stage of semantic analysis, as described in the methods. After back translation, the ATDM-FEV version was compared with the original ATDM-OV version by lead author, Dr. John Piette, who gave his agreement, and the final Brazilian Portuguese version was completed. The cultural adaptation process lasted three months.

As mentioned, the semantic analysis had five people with DM involved. In analyzing possible changes, additions and suggestions of the participants, no changes were suggested to the ATDM-PCV2 instrument. In the same way, the ATDM-PCV2 instrument was also applied to five persons with DM, and no modifications in completion and understanding of the instrument through pretest application was needed.

The title of the instrument Automated Telephone Disease Management (ATDM) Satisfaction Scales was then named Escala de Satisfação para Manejo da Doença Automatizado por Telefone (MDAT)

As previously stated, 39 Brazilian adults were involved in the reliability analysis, with no refusals. The average age was 60, the majority were female (67.4%); 65.2% were married, 51.2% retired, 60.5% attended school up to elementary school, with an average of 7 years of study. Regarding family income, 44.1% reported income three to four times the minimum wage.

With respect to the Satisfaction Scale items for MDAT, in the first item, 94.9% answered that the words used during the connections were always easy to understand. As to the question regarding the sound volume of the voice calls, 36 (92.3%) responded that the sound level was always enough to hear without difficulty. When asked if the information was given too fast, for more than half, 53.8%, the answer was never. Twenty-eight (71.8%) reported never having difficulty responding to questions using a phone call.

The majority (64.1%) stated that the phone calls made them assured that the nurse knew how they were and 51.3% said they always learned something new during the calls; 69.2% said the calls always reminded them to do something, how to check their blood sugar or eat healthy foods.

The majority (94.9%) responded that the calls were interesting; 92.3% always enjoyed receiving the calls, and 87.2% said that the calls were never uncomfortable. On the last item, 84.6% said that the length of calls was appropriate.

The analysis of reliability for internal consistency of the items of the adapted version was calculated by Cronbach's alpha coefficient, resulting in α = 0.39 for the total scale, ranging between values of 0.30 and 0.43 for the 11 items of the instrument.

DISCUSSION

The Brazilian health system needs to be strengthened to provide assistance to people with NCDs, through: models of care for chronic conditions based on local experiences; expansion and upgrading of the Family Health Strategy; increased access to cost-effective medicines; better communication between basic attention and other levels of care; integration of programmatic actions for chronic diseases, among other things ⁽²³⁾.

In this context, along with the modernization and integration of new technologies in health care, it is necessary to evaluate the use of these new technologies, so that health actions are more efficacious, efficient and easily accessible to the entire population. Thus, to understand and evaluate the care provided to people with diabetes mellitus, it is necessary to use new technologies, with the phone being a very advantageous option.

The assessment of health technology needs to be expanded to provide a strong basis for appropriate selection of new programs and public health actions, and new medications, devices and diagnostic tests ⁽²³⁾. An evaluation consists of making a value judgment about an intervention, in order to assist in decision making ⁽²⁴⁾.

Thus, the introduction of methodologies such as assessment of satisfaction of persons with diabetes mellitus, after participating in telephone educational programs by a health system, gives a comprehensive knowledge of the needs of the person, whereas the lack of measuring instruments in the area are aspects to be considered in studies that evaluate the effect of new treatment modalities and what impact they can bring to health care, in addition to expected outcome data.

The mean age was 60 years. Most participants were women with low educational levels. These findings are in line with the profile of the sample of studies that compared the effectiveness of educational strategies in a Diabetes Education Program ^(17,18).

The predominance of females is indicative of women presenting higher searching behavior for self-care, and are more assiduous in regards to educational programs than men ⁽¹⁹⁾. The level of formal education is an important feature to consider in proposing educational programs, because the lower education may hinder access to information and impair treatment adherence ⁽²⁰⁾.

Regarding the occupation and the prevalence of retired persons, the study showed that only 25% of older people were economically active. People who worked showed greater physical and mental disposition, higher educational level and family income, and lower prevalence of chronic diseases ⁽²¹⁾. Low socioeconomic status found in this research should be taken in consideration, since the person who stays active can achieve greater personal satisfaction, opportunity for social interaction and benefits to physical and mental health ⁽²²⁾.

After finalizing the stages of cultural adaptation, people with diabetes mellitus answered the Satisfaction Scale for MDAT easily and quickly, it was found that there was no difficulty in understanding of the issues as to the adequacy of the response categories, which were used easily.

For the analysis of internal consistency of the items of the adapted version, the Cronbach alpha coefficient was calculated, with values of α = 0.39 for the total scale, ranging between 0.30 and 0.43 for the items. However, 0.70 is considered an ideal minimum value, and 0.60 for exploratory research may be accepted ⁽²⁵⁾. We justified the low values found for measures of the items in this study as a limitation of the sample size. It is clarified that this is an early stage of the validation process of the instrument, considering the small sample size for the amount of items of the adapted instrument. It is important to highlight that the study sample will be increased to assess the psychometric properties of the adapted Brazilian instrument.

It is important to consider that in a country of large expanse, like Brazil, different cultural contexts can be identified and it may have implications for adaptation and validation of instruments to measure specific constructs, such as the satisfaction of people after beeing submitted to educational programs by phone. Thus, Brazilian adults with DM may be developing in different cultural and social contexts within the same nation. Low educational levels found in the sample of this study added to the social and cultural context of people with chronic conditions, as an important source of knowledge acquisition and modification of knowledge ⁽²⁶⁾ are reasons that may be influenced by different cultures, lifestyles and education.

CONCLUSION

The ATDM Satisfaction Scales instrument , originally in English, was translated and culturally adapted into Brazilian Portuguese, following all the steps listed in sequential methodology.

Thus, it was observed that the idiomatic, semantic, cultural and conceptual equivalence of the original instrument have been retained, and the Satisfaction Scale for Automated Telephone Disease Management, translated and adapted for the Brazilian context, maintained the concepts and the evaluation of the dimensions of the original instrument, including being ratified by the author.

With respect to those results achieved by internal consistency analysis, it was found that the adapted instrument presented low levels, since this is an initial phase of the study with a small sample size. However, the sample will be further expanded in the same social environment of the study site, in order to enable a workable reliability of the instrument and additional statistical analyses.

It is expected that after the replication of this study in a larger sample size and analysis of the psychometric properties of the Satisfaction Scale for MDAT in Brazil, this instrument can then be used by Brazilian researchers, and their results can be compared with other cultures, as well as incorporated as an additional tool in the daily care of health professionals to monitor health status over time and, thereby to know the impact of their interventions on the condition and evolution of Brazilian adults with DM.

Given the above, we reiterate the importance for the development of studies of this nature that may contribute to the understanding of the factors involved with the satisfaction of people with DM after participating in educational programs conducted by telephone, relevant within the context of managed care, ensuring therefore the continuity of health interventions and contributing to clinical practice and nursing education.

REFERENCES

• ¹
  World Health Organization. World Health Report 2002. Reducing risks, promoting healthy life. Geneva: WHO; 2002.
• 2. International Diabetes Federation. Complicações do diabetes e educação. Diabetes Clín. 2004; 6(3):217-20.
• 3. The effect of intensive treatment of diabetes on the development and progression of the long-term complications in insulin-dependent diabetes Mellitus. The Diabetes Control and Complications Trial Research Group. N Engl J Med. 1993; 329(14):977-86.
• 4. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). UK Prospective Diabetes Study (UKPDS) Group. Lancet. 1998; 352(9131):837-53.
• 5. Blake H. Innovation in practice: mobile phone technology in patient care. Br J Community Nurs. 2008; 13(4):160, 162-5.
• 6. Peck A. Changing the face of standard nursing practice through telehealth and Telenursing. Nurs Adm Q. 2005; 29(4):339-43.
• 7. Snooks HA, Williams AM, Griffiths LJ, Peconi J, Rance J, Snelgrove S, et al. Real nursing? The development of telenursing. J Adv Nurs. 2008; 61(6):631-40.
• ⁸
  Brasil. Ministério da Saúde. VIGITEL Brasil 2006. Vigilância de fatores de risco e proteção para doenças crônicas por inquérito telefônico: estimativas sobre frequência e distribuição sócio-demográfica de fatores de risco e proteção para doenças crônicas nas capitais dos 26 estados brasileiros e no Distrito Federal em 2006. Brasília: Ministério da Saúde; 2007.
• 9. Kim HS, Oh JA. Adherence to diabetes control recommendations: impact of nurse telephone calls. J Adv Nurs. 2003; 44(3):25661.
• 10. Zanetti ML, Otero LM, Biaggi MV, dos Santos MA, Péres DS, Guimarães FP. Satisfaction of diabetes patients under follow-up in a diabetes education program. Rev Latinoam Enferm. 2007; 15(4):583-9.
• 11. Piette JD. Satisfaction with automated telephone disease management calls and its relationship to their use. Diabetes Educ. 2000; 26(6):1003-10.
• 12. Kerlinger FN. Metodologia da pesquisa em ciências sociais. São Paulo: EPU; 1979.
• 13. Sapnas KG, Zeller RA. Minimizing sample size when using exploratory factor analysis for measurement. J Nurs Meas. 2002; 10(2);135-54.
• 14. Belle GV, Fisher LD, Heagerty PJ, Lumley T. Biostatistics: a methodology for the health sciences. 2nd ed. New Jersey: John Wiley & Sons; 2004.
• 15. Ferrer M, Alonso J, Prieto L, Plaza V, Monsó E, Marrades R, et al. Validity and reliability of the St George's Respiratory Questionnaire after adaptation to a different language and culture: the Spanish example. Eur Respir J. 1996;9(6):1160-6.
• 16. Cummings SR, Stuart A, Hulley SB. Elaboração de questionários e instrumentos de coleta de dados. In: Hulley SB, Cummings SR, Browner WS, Grady D, Hearst N, Newman TB. Delineando a pesquisa clínica: uma abordagem epidemiológica. 2a ed. Porto Alegre: Artmed; 2003. p. 265-81.
• 17. Torres HC, Franco LJ, Stradioto MA, Hortale VA, Schall VT. Evaluation of a diabetes education program. Rev Saude Publica. 2009; 43(2):2918.
• 18. Zanetti ML, Otero LM, Peres DS, dos Santos MA, Guimarães FP, Freitas MC. Progress of the patients with diabetes mellitus who were managed with the staged diabetes management framework. Acta Paul Enferm. 2007; 20(3):338-44.
• 19. Péres DS, Franco LJ, dos Santos MA. Feelings of women after the diagnosis of type 2 diabetes. Rev Latinoam Enferm. 2008; 16(1):1018.
• 20. Delamater AM. Improving patient adherence. Clin Diabetes. 2006; 24(2):717.
• 21. Giatti L, Barreto SM. [Health, work and aging in Brazil]. Cad Saude Publica. 2003; 19(3): 75971. Portuguese
• 22. Kinder LS, Kamarck TW, Baum A, Orchard TJ. Depressive symptomatology and coronary heart disease in Type 1 diabetes mellitus: a study of possible mechanisms. Health Psychol. 2002; 21(6):54252.
• 23. Schmidt MI, Ducan BB, Azevedo e Silva G, Menezes AM, Monteiro CA, Barreto SM, Chor D, Menezes PR. Chronic non-communicable diseases in Brazil: burden and current challenges. Lancet. 2011; 377(9781):1949-61.
• 24. Contadriopoulos AP, Champagne F, Denis JL, Pineault R. A avaliação na área da saúde: conceitos e métodos. In: Hartz ZM, organizador. Avaliação em saúde: dos modelos conceituais à prática na análise de implantação de programas. Rio de Janeiro: Fiocruz; 1997. p. 29-48.
• 25. Pestana MH, Gageiro JG. Análise de dados para ciências sociais: a complementaridade do SPSS. 3a ed. Lisboa: Silabo; 2003.
• 26. Duarte S. Saber de saúde e de doença: porque vão as pessoas ao médico. Coimbra: Quarteto editores; 2002.

*

Trabalho de conclusão do curso de Graduação em Bacharelado em Enfermagem, apresentado à Escola de Enfermagem de Ribeirão Preto – Universidade de São Paulo – USP – Ribeirão Preto (SP), Brasil, 2011. Suporte Financeiro: Fundação de Amparo à Pesquisa do Estado de São Paulo – FAPESP – Iniciação Científica – IC – processo nº 2010/07522-4.

Publication Dates

• Publication in this collection
  02 Oct 2012
• Date of issue
  2012

History

• Received
  08 Nov 2011
• Accepted
  08 Apr 2012