Acessibilidade / Reportar erro

Cross-cultural adaptation of the Readiness for Interprofessional Learning Scale in Brazil

ABSTRACT

Objective

Conduct a cross-cultural adaptation of the expanded version of the 29-items Readiness for Interprofessional Learning Scale (RIPLS) into Brazilian Portuguese.

Method

Five steps were adopted: three translations, synthesis, three back-translations, assessment by an expert committee, and pre-test. Validation comprised 327 students from 13 undergraduate health courses from a public university. Parallel analyses were conducted using the R software and factor analysis using Exploratory Structural Equation Modeling.

Results

11. Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes (update). Cochrane Database Syst Rev. 2013;(3):CD002213. 99. Thannhauser J, Russel-Mayhew S, Scott C. Measures of interprofessional education and collaboration. J Interprof Care. 2010;24(4):336-49. 1212. Wilhelmsson M, Ponzer S, Dahlgren LO, Timpka T, Faresjö T. Are female studentes in general and nursing students more ready for teamwork and interprofessional collaboration in healthcare? BMC Med Educ. 2011;11:15. doi 10.1186/1472-6920-11-15 1616. El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32. 1010. Coster S, Norman I, Murrells T, Kitchen S, Meerabeau E, Sooboodoo E, et al. Interprofessional attitudes among undergraduate students in the health professions: a longitudinal questionnaire survey. Int J Nurs Stud. 2008;45(11):1667-81. 1111. Aziz Z, Teck LC, Yen PY. The attitudes of medical, nursing and pharmacy studentes to inter-professional learning. Soc Behav Sci. 2011;29:639-45. 1717. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25(24):3186-91. 1919. Hair JF, Anderson RE, Tathan RL, Black WC. Análise multivariada de dados. 5ª ed. Porto Alegre: Bookman; 2005. 2121. Venables WN, Smith DM. The R Development Core Team. An introduction to R, Version 2.2.0 R-Project [Internet]. [cited 2013 Oct 10]. Available from: http://CRAN.R-project.org
http://CRAN.R-project.org...
2424. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. 7nd ed. New Jersey: Pearson Education International; 2010. 2525. Razali NM, Wah YB. Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. J Stat Modeling Anal. 2011;2(1):21-33. 2929. Williams B, Brown T, Boyle M. Construct validation of the readiness for interprofessional learning scale: a rasch and factor analysis. J Interprof Care 2012;26(4):326-32.

Conclusion

Evidences were found relating to the validity of the RIPLS version in Brazilian Portuguese in its application in the national context.

Interprofessional Relation; Education; Methods; Questionnaire

RESUMO

Objetivo

Realizar a adaptação transcultural e a validação da versão de 29-itens daReadiness for Interprofessional Learning Scale (RIPLS) para língua portuguesa falada no Brasil.

Método

Foram adotadas cinco etapas: três traduções, síntese, três retrotraduções, avaliação por especialistas e pré-teste. A validação contou com 327 estudantes de 13 cursos de graduação de uma universidade pública. Foram realizadas análises paralelas com o software R e a análise fatorial utilizando Modelagem de Equações Estruturais.

Resultados

A análise fatorial resultou em uma escala de 27 itens e três fatores: Fator 1 – Trabalho em equipe e colaboração com 14 itens (1-9, 12-16), Fator 2 – Identidade profissional, oito itens (10, 11, 17, 19, 21-24), e Fator 3 – Atenção à saúde centrada no paciente, cinco itens (25-29). Alfa de Cronbach dos três fatores foi respectivamente: 0,90; 0,66; 0,75. Análise de variância mostrou diferenças significativas nas médias dos fatores dos grupos profissionais.

Conclusão

Foram identificadas evidências de validação da versão em português da RIPLS em sua aplicação no contexto nacional.

Relações Interprofissionais; Educação Superior; Questionários; Comparação Transcultural; Estudos de Validação

RESUMEN

Objetivo

Realizar la adaptación transcultural y la validación de la versión de 29 ítems de la Readiness for Interprofessional Learning Scale(RIPLS) para el portugués hablado en Brasil.

Método

Adoptado cinco etapas: tres traducciones, síntesis, tres retro-traducciones, evaluación de expertos y pretest. Validación consistió de 327 alumnos de 13 cursos de formación de grado en una universidad pública. Análisis paralelos con el software R y análisis factorial utilizando Modelo de Ecuaciones Estructurales fueron realizados.

Resultados

El análisis factorial resultó en una escala de 27 ítems y tres factores: Factor 1 - Trabajo en equipo y colaboración con 14 ítems (1-9, 12-16), Factor 2 - Identidad Profesional ocho ítems (10, 11, 17, 19, 21-24) y Factor 3 - Atención a la salud centrada en el paciente, cinco ítems (25- 29). El Alfa de Cronbach de los tres factores fueron, respectivamente: 0,90; 0,66; 0.75. Análisis de varianza mostró diferencias significativas en los promedios de los grupos profesionales.

Conclusión

Se identificaron evidencias de validación de la versión en portugués de RIPLS en su aplicación en el contexto nacional.

Relaciones Interprofesionales; Educación Superior; Cuestionarios; Comparación Transcultural; Estudios de Validación

INTRODUCTION

Interprofessional education (IPE) has been defined as any intervention “when members of more than one health or social care (or both) professions learn interactively together, for the explicit purpose of improving interprofessional collaboration or the health/well being (or both) of patient/clients”(11. Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes (update). Cochrane Database Syst Rev. 2013;(3):CD002213. p.5). IPE programmes have been delivered internationally for decades now and are considered to be an evidence-based approach to broadening healthcare-related service responsiveness and quality(22. World Health Organization. Framework for action on interprofessional education & collaborative practice. Geneva: WHO; 2010.). However, although IPE is relatively common in countries such as the USA, Canada and the UK, it has only recently been introduced in Brazil(33. Batista NA. Educação interprofissional em saúde: concepções e práticas. Cad FNEPAS. 2012;(2):25-8.,44. Peduzzi M, Norman I, Germani ACGG, Silva JAM, Souza GC. Interprofessional education: training for healthcare professionals for teamwork focusing on users. Rev Esc Enferm USP [Internet]. 2013 [cited 2014 Oct 10];47(4):977-83. Available from: http://www.scielo.br/pdf/reeusp/v47n4/en_0080-6234-reeusp-47-4-0977.pdf
http://www.scielo.br/pdf/reeusp/v47n4/en...
).

Historically, the predominant model of Brazilian higher health education was uniprofessional and biomedical in its orientation(55. Feuerwerker LCM, Sena RR. Contribuição ao movimento de mudança na formação profissional em saúde: uma avaliação das experiências UNI. Interface Comun Saúde Educ. 2002;6(10):37-50.). Notably, in the 1990’s, the introduction of the UNI program - ‘A New Initiative in the Education of Health Professionals: Union to the Community’- helped to promote the concept of interdisciplinarity and multiprofessional practice(55. Feuerwerker LCM, Sena RR. Contribuição ao movimento de mudança na formação profissional em saúde: uma avaliação das experiências UNI. Interface Comun Saúde Educ. 2002;6(10):37-50.). In 2000 the Brazilian National Curriculum Guidelines (BNCG) for Undergraduate Courses in Health was implemented, which emphasized teaching based on learning competencies, including multidisciplinary team working, to all health courses. The BNCG initiative resulted several innovative educational programmes with partially integrated curricular programs for a range of undergraduates from medicine, nursing, dentistry, psychology, pharmacy and physiotherapy(66. Barreto ICHC, Loiola FA, Andreade LOM, Moreira AEMM, Cavalcanti CGCS, Arruda CAM, et al. Development of interprofessional collaborative practices within undergraduate programs on healthcare: case study on the Family Health Alliance in Fortaleza (Ceará, Brazil). Interface Comum Saúde Educ. 2011;15(36):199-212.). These initiatives have had the support of a range of important stakeholders, including the Ministry of Health, Pan-American Health Organization and the World Health Organization, and other international agencies such as the W.K. Kellogg Foundation, along with the Universities themselves(66. Barreto ICHC, Loiola FA, Andreade LOM, Moreira AEMM, Cavalcanti CGCS, Arruda CAM, et al. Development of interprofessional collaborative practices within undergraduate programs on healthcare: case study on the Family Health Alliance in Fortaleza (Ceará, Brazil). Interface Comum Saúde Educ. 2011;15(36):199-212.). Despite these advances, there remains continuing debate about the merits of introducing interprofessional training in Brazil.

To date student formal and informal educational opportunities for IPE for Brazilian are rare(33. Batista NA. Educação interprofissional em saúde: concepções e práticas. Cad FNEPAS. 2012;(2):25-8.). For example, among 197 undergraduate medical courses (42% public and 58% private) and 883 undergraduate nursing courses (16% public and 845 private)(77. Brasil. Ministério da Educação. Instituições de Educação Superior e Cursos Cadastrados [Internet]. Brasília; 2012-2013 [citado 18 fev. 2013]. Disponível em: http://emec.mec.gov.br/
http://emec.mec.gov.br/...
), only one has developed an integrated curriculum based on IPE approach, and only a handful have developed independent courses which promote interprofessional practice(1111. Aziz Z, Teck LC, Yen PY. The attitudes of medical, nursing and pharmacy studentes to inter-professional learning. Soc Behav Sci. 2011;29:639-45.). However, this is an imperative for IPE to develop in Brazil, given the relevance of teamwork to public health policy and the strengthening of the Unified Health System(66. Barreto ICHC, Loiola FA, Andreade LOM, Moreira AEMM, Cavalcanti CGCS, Arruda CAM, et al. Development of interprofessional collaborative practices within undergraduate programs on healthcare: case study on the Family Health Alliance in Fortaleza (Ceará, Brazil). Interface Comum Saúde Educ. 2011;15(36):199-212.).

In order to determine the optimum environment for effective interprofessional learning in Brazil, it is necessary to focus research on what kind of education works, under what circumstances and how this might impact on outcomes for staff members, patients and the larger organization. One of the most important contextual variables to consider when introducing IPE or interprofessional learning (IPL) is the prior attitudes and expectations of healthcare students who may not have learned or worked together before. The Readiness for Interprofessional Learning Scale (RIPLS)(88. Parsell G, Bligh J. The development of a questionnaire to assess the readiness of health care students for interprofessional learning (RIPLS). Med Educ. 1999;33(2): 95-100.) is a validated and widely employed instrument which gathers assesses the readiness of students to participate with other students in shared learning(99. Thannhauser J, Russel-Mayhew S, Scott C. Measures of interprofessional education and collaboration. J Interprof Care. 2010;24(4):336-49.). The scale has been used in several recent studies to investigate the attitudes of healthcare students and professionals(1010. Coster S, Norman I, Murrells T, Kitchen S, Meerabeau E, Sooboodoo E, et al. Interprofessional attitudes among undergraduate students in the health professions: a longitudinal questionnaire survey. Int J Nurs Stud. 2008;45(11):1667-81.-1111. Aziz Z, Teck LC, Yen PY. The attitudes of medical, nursing and pharmacy studentes to inter-professional learning. Soc Behav Sci. 2011;29:639-45.) and the impact of the characteristics of students and education programs toward cooperating with colleagues from other professional groups(1212. Wilhelmsson M, Ponzer S, Dahlgren LO, Timpka T, Faresjö T. Are female studentes in general and nursing students more ready for teamwork and interprofessional collaboration in healthcare? BMC Med Educ. 2011;11:15. doi 10.1186/1472-6920-11-15). Recent studies to compare psychometric properties of RIPLS and IEPS (Interdisciplinary Education Perception Scale), have suggested that both these scales can be used to track the development and impact of IPE education in students(1313. Lie DA, Fung CC, Trial J, Lohenry K. A comparison of two scales for assessing health professional student’s attitude toward interprofessional learning. Med Educ Online 2013;18:21885. doi: 10.3402/meo.v18i0.21885.). The first published version of the RIPLS scale(88. Parsell G, Bligh J. The development of a questionnaire to assess the readiness of health care students for interprofessional learning (RIPLS). Med Educ. 1999;33(2): 95-100.) had 19 items and comprised three factors (Teamwork and collaboration, Professional identity, Roles and responsibilities). More recently, the authors inserted ten new items resulting in a 29 item version, comprising four factors (Teamwork and collaboration, Professional identity, Role and responsibilities, and a new factor relating to Patient-centeredness)(1414. Mattick K, Bligh J. Readinss for interprofessional learning scale. In: Bluteau J, editor. Interprofessional education. London: MacMillan; 2007.) The expanded version of the 29-item RIPLS was validated by Reid et al(1515. Reid R, Bruce D, Allstaff K, McLernon D. Validating the Readiness for Interprofessional Learning Scale (RIPLS) in the postgraduate context: are health care professionals ready for IPL? Med Educ. 2006;40(5):415-22.). with postgraduate professional: general practitioners, nurses, pharmacists and allied health professional in the local primary care organization. It was also validated by El-Zubeir et al(1616. El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32.). with medical and nurses students in their last two years of education in the United Arab Emirates. Given that there are currently no interprofessional attitudes scale suitable for use in populations within Brazil, the purpose of this paper was to develop a version of the 29 item of the RIPLS into Portuguese spoken in Brazil.

METHOD

Phase 1: The cross-cultural adaptation of the RIPLS to the portuguese language

Stages 1-3: Translation of the instrument

This paper describes a two-phase cross-cultural adaptation of the RIPLS into Brazilian Portuguese, guided by a specific methodology comprising validation(1717. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25(24):3186-91.). The cross-cultural adaptation phase highlights five essential stages: translation, synthesis, back-translation, assessment by an experts committee, and pre-test(1717. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25(24):3186-91.).

In stage 1, three independent translations were made by professional translators whose native language was Brazilian Portuguese. This was followed by stage 2, where a synthesis of this translation was conducted by the researcher to produce a single common translation. In stage 3, three back-translations were produced by native speakers of English translators who were not acquainted with the original questionnaire, but who had proficiency in Portuguese.

Stage 4: assessment by the experts committee

Content validity is often determined through qualitative expert panel but quantitative analysis of reviewers’ agreement is also suggested in the literature(1818. Wynd CA, Schmidt B, Schaefer MA. Two quantitative approaches for estimating content validity. West J Nurs Res. 2003;25(5):508-18.). Authors advocates the use of an agreement index which allows for the quantitative verification of the content’s validation. To complete this stage of the study, a committee of eleven experts was convened comprising academics from a range of undergraduate courses including medicine, nursing, speech therapy, occupational therapy and two experts in the methodology of cross-cultural adaptation of scales. These experts were handed copies of the translations, syntheses and back-translations, as well as the specific instrument and recorded their individual assessment concerning the equivalence (semantic, idiomatic, experimental and cultural) of the synthesized version and the original English version. This group assessment employed criteria where being having agreement equal to or higher than 90% was regarded as fair whilst being lower than 80% was considered as only adequate and required further discussion at a subsequent meeting. Twenty four scale items reached 80% agreements, while five items reached lower scores of agreements ranging from 72.7% (items 5, 6 and 25) to 54.5% for item 17 (“The function of nurses and therapists is mainly to serve as a support to doctors”). The experts discussed and reworded items in order to reach a consensus higher than 90%. At the end of the assessment process, the experts committee reached 90% agreement for all 29 items of the scale, thus producing the pre-final version of the adapted scale.

Stage 5: Pre-test

In the final stage 5, a pre-test was carried out in March 2011 with 54 undergraduate students of the second-year and third-year of medicine, nursing, nutrition, occupational therapy, physiotherapy and speech therapy at a public university in São Paulo, Brazil. In order to approach the students, their professors were asked for their permission to and required to present the research either at the beginning or the end of the class. The research was approved by the Ethics Research Committee (Process 919/2010/CEP) (two-phase – cross-cultural adaptation and validation), and authorized by the directors of the thirteen education units involved. The sample was not stratified for the purposes of the pre-test.

After self-administration, each student was interviewed to determine any problems with the scale or individual items, and whether further modifications therefore needed to be made before validation. Among the 54 students included in the pre-test sample, 35 (64.8%) also agreed to participate in a post test interview. Of these, 17 (48.5%) had no comprehension difficulty with any of the items, whilst four (11.4%) had problems with just one item from the following: 2, 5, 6 or 19; two (5.7%) with item 18; three (8.6%) with item 22; and 3 (8.6%) with item 27. Five (14.2%) students also had comprehension difficulty with item 20.

Phase 2: Validation of the portuguese version of the ripls within the brazilian context

Sample

The sample comprised 347 third-year undergraduate students from the public university that took part in the validation study by completing the final version of the translated scale. This sample excluded the 54 students of the pre-test. At the time of administration neither students in the pre-test nor the validation study had participated in any formal interprofessional education at the University. Students were included Physical activity sciences, physical education, nursing, pharmacy, physiotherapy, speech therapy, gerontology, medicine, nutrition, obstetrics, dentistry, psychology, and occupational therapy.

The sample size was generated by employing the minimum criteria of 10 participants per each item of the scale(1919. Hair JF, Anderson RE, Tathan RL, Black WC. Análise multivariada de dados. 5ª ed. Porto Alegre: Bookman; 2005.), therefore requiring up to 290 students, plus 20% of expected sample loss, thus resulting in 348 participants, with one subsequently lost to follow up. The sample was stratified by course, from the total amount of students in the second period of 2011, which amounted 901. To define the stratified sample it was used a sampling fraction of 0,386238 that resulted from the total sample size of 348 divided by the total of 901 students This sampling fraction was later applied to each of the student’s courses to make a stratified sample (Table 1)

Table 1
Number of third-year students and size of the sample, São Paulo 2011-2012.

Data collection was performed between October 2011 and May 2012. In order to approach the students, their third-year professors (second semester of 2011 and first semester of 2012) were asked for their permission to and required to present the research either at the beginning or the end of the class. In a date previously set with the professors, field researchers that were trained to the study, oriented students on the research and invited them to take part in the study.

Statistical analysis

To evaluate the RIPLS dimensionality parallel analyses were performed(2020. Horn JL. A rationale and technique for estimating the number of factors in factor analysis. Psychometrika. 1965;30:179-85.)using R software(2121. Venables WN, Smith DM. The R Development Core Team. An introduction to R, Version 2.2.0 R-Project [Internet]. [cited 2013 Oct 10]. Available from: http://CRAN.R-project.org
http://CRAN.R-project.org...
), to elicit information about the number of factors to be extracted. The dimensionality of the scale was evaluated using the Exploratory Structural Equation Modeling ESEM(2222. Asparouhov T, Muthén B. Exploratory structural equation modeling. Struct Equ Modeling. 2009;16(4):397-438.)running software Mplus 7.

As the Likert scale used in the RIPLS has an ordinal categorical level of measurement, polichoric correlation matrix was chosen as the most suitable source of information for the ESEM analyses. The estimation method was theweighted least squares means and variance (WLSMV) as recommended for models with categorical indicators that do not show a multivariate normal distribution, as in the case of the RIPLS indicators. An oblique GEOMIN rotation was performed in the data matrix to estimate the correlations among the extracted factors(2222. Asparouhov T, Muthén B. Exploratory structural equation modeling. Struct Equ Modeling. 2009;16(4):397-438.).

Based on literature, the following parameters were used for deciding about the fitness of the factorial model: Comparative Fit Index (CFI) [values ≥0,90 show acceptable fit, expecting ≥0,95 for fit]; root mean square error approximation (RMSEA) [values≤0,06 meaning fit, with a maximum acceptable limit of 0,08] and Weighted Root Mean Square Residual (WRMR) with values less than 0,90 meaning fit models(2323. Muthén LK, Muthén BO. Mplus user’s guide. 2nd ed. Los Angeles: Muthén & Muthén; 2001., 2424. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. 7nd ed. New Jersey: Pearson Education International; 2010.). Due to the fact of being an exploratory study a value of 0,60 was defined as the lower acceptable limit for the internal consistency index Cronbach´s Alpha(1919. Hair JF, Anderson RE, Tathan RL, Black WC. Análise multivariada de dados. 5ª ed. Porto Alegre: Bookman; 2005.). Analysis of Variance (ANOVA) was performed to verify if there were significant differences in the average of the factors depending on the participants´ training course. Post-hoc analyses used Tukey’s honest significance difference test (HSD)(1919. Hair JF, Anderson RE, Tathan RL, Black WC. Análise multivariada de dados. 5ª ed. Porto Alegre: Bookman; 2005.).

RESULTS

Firstly, in the validation phase of the study, an exploratory analysis was performed on the dataset to verify the assumptions of the general linear model. Missing Values Analysis showed a low percentage of missing data (0,9 %), all of them non systematic. Mahalanobis Distance test showed 20 multivariate outliers (critical boundary=56,892; p<0,001; df=28) representing 5.76% of the whole sample (n=347). Those outliers proved to be non systematic and were excluded from subsequent analyses, that were performed on the remaining 327 cases.

Shapiro-Wilk normality tests showed that the item distribution was different from the normal distribution at a p<0,001 level(2525. Razali NM, Wah YB. Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. J Stat Modeling Anal. 2011;2(1):21-33.). However, the graphic analysis of normal probability showed that the difference was just marginal and WLSMV can accommodate small differences from normality, especially when used in samples over 200 cases(2424. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. 7nd ed. New Jersey: Pearson Education International; 2010.).

Analysis of scale dimensionality - Exploratory factor analysis

The results of the parallel analyses(2020. Horn JL. A rationale and technique for estimating the number of factors in factor analysis. Psychometrika. 1965;30:179-85.) indicated the pertinence of extracting up to three factors as shown in Figure 1.

Figure 1
The results of the parallel analysis.

The factor solution comprising three factors with the WLSMV method showed fitness regarding data (CMIN=576,860; df=322; p<0,001; CFI=0,97; RMSEA=0,049 CI90%: 0,043-0,056; PCLOSE: 0,573; WRMR=0,830). All the items presented significant factor loadings at the p<0,005 level. Only four items showed saturation loading ≥0,30 in more than one factor (items 10, 11, 12 and 24), and that number of complex items may be considered relatively low. Two additional items did not present saturation loading in any factor, being therefore excluded from the instrument (items 18 and 20). The Table 2 details the factor matrix for the Brazilian Portuguese of the RIPLS, with the validity and reliability indicators.

Table 2
Distribution of the items loading and the Cronbach’s Alpha of the three factors in the validated Brazilian Portuguese version of the RIPLS, Sao Paulo – 2011.

The Brazilian Portuguese version of the RIPLS was thus composed by 27 items, grouped in three factors. The first factor ‘Teamwork and Collaboration’, with 14 items (1-9 and 12-16) related to positive attitudes/ availability for joint learning, trust, respect, collaboration and teamwork among health professions students . The second factor ‘Professional Identity’ was composed of by 8 items (1010. Coster S, Norman I, Murrells T, Kitchen S, Meerabeau E, Sooboodoo E, et al. Interprofessional attitudes among undergraduate students in the health professions: a longitudinal questionnaire survey. Int J Nurs Stud. 2008;45(11):1667-81., 1111. Aziz Z, Teck LC, Yen PY. The attitudes of medical, nursing and pharmacy studentes to inter-professional learning. Soc Behav Sci. 2011;29:639-45., 1717. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25(24):3186-91., 1919. Hair JF, Anderson RE, Tathan RL, Black WC. Análise multivariada de dados. 5ª ed. Porto Alegre: Bookman; 2005., 2121. Venables WN, Smith DM. The R Development Core Team. An introduction to R, Version 2.2.0 R-Project [Internet]. [cited 2013 Oct 10]. Available from: http://CRAN.R-project.org
http://CRAN.R-project.org...

22. Asparouhov T, Muthén B. Exploratory structural equation modeling. Struct Equ Modeling. 2009;16(4):397-438.

23. Muthén LK, Muthén BO. Mplus user’s guide. 2nd ed. Los Angeles: Muthén & Muthén; 2001.
-2424. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. 7nd ed. New Jersey: Pearson Education International; 2010.). From those, 5 items are related to negative attitudes and unavailability for interprofessional learning. The remaining three items (2222. Asparouhov T, Muthén B. Exploratory structural equation modeling. Struct Equ Modeling. 2009;16(4):397-438.

23. Muthén LK, Muthén BO. Mplus user’s guide. 2nd ed. Los Angeles: Muthén & Muthén; 2001.
-2424. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. 7nd ed. New Jersey: Pearson Education International; 2010.) refer to professional autonomy and clinical objectives of each profession. In this sense this factor expresses the attitude regarding professional identity, albeit with a competitive component. The third factor ‘Patient Centeredness’ with 5 items (2525. Razali NM, Wah YB. Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. J Stat Modeling Anal. 2011;2(1):21-33.

26. McFadyen AK, Webster VS, Strachan K, Figgins E, Brown H, McKechnie J. The Readiness for Interprofessional Learning Scale: a possible more stable sub-scale model for the original version of RIPLS. J Interprof. 2005;19(6):595-603.

27. Pasquali L. Análise fatorial para pesquisadores. Brasília: LabPam; 2005.

28. Tamura Y, Seki K, Usami M, Taku S, Bontje P, Ando H, et al. Cultural adaptation and validating a Japanese version of the Readiness for Interprofessional Learning Scale (RIPLS). J Interprof Care. 2012;26(1):56-63.
-2929. Williams B, Brown T, Boyle M. Construct validation of the readiness for interprofessional learning scale: a rasch and factor analysis. J Interprof Care 2012;26(4):326-32.) deals with positive attitudes/ availability to understand the health needs from the patient’s perspective, in a relation based on trust, compassion and cooperation.

From a structural point of view, these three factors are correlated. There is a pattern of positive relation between Factor 1 Teamwork and Collaboration and factor 3 Patient Centeredness (r=0,44), showing a directly proportional relationship. Inversely, the association pattern of Factor 2 Professional Identity with the rest of the factors was negative F2 – F1 (r=-0,53); F2 - F3 (r=-0,33). To analyses the scores and representation of the average points in the three factors, all items were reversed. In this way, the scale of answers represented the following numbers/ semantic labels (1=Totally disagree, 2=Disagree, 3=Do not agree or disagree, 4=Agree, 5=Totally agree). Thus the larger the score, the larger the concordance with the item, and the stronger the attitudes/readiness for interprofessional learning. The correlation found was the following: F1 and F2 (r=-0,45;p<0,001); F1 and F3 (r=0,35;p<0,01); F2 and F3 (r=-0,05;p<0,38). The relationship between “Professional Identity” and “Patient Centeredness” was not significant.

Analysis of variance of students in the different professional groups

Levene´s homogeneity of variance tests showed a homoscedastic pattern for the three factorial scores (F1. Levene=0,784; p<0,667; F2. Levene=1.377; p<0,175; F3. Levene=1,402; p<0,164). ANOVA results showed significant differences in the level of all factor scores: F1 [F(2,326)=5,413, p≤0,001]; F2 [F(2,326)=14,233, p≤0,001]; F3 [F(2,326)=2,365, p≤0,006]. Post hoc analysis showed that in the first factor of “Teamwork and collaboration” Medical students had significantly lower averages than Speech Therapy, Nutrition and Gerontology students. No other significant differences were found in other courses´ students. In the second factor, “Professional Identity” four sub-groups with significant difference were formed, but in a general fashion all sub-groups showed low scores, thus suggesting low concordance with the variables of F2 that are related to competitive attitude against the other professions. It is remarkable that the sub-group 1 (Gerontology, Nursing, Physical Education, and Occupational Therapy) has a higher disagreement with this factor, while sub-group 4 (Medicine, Dentistry and Physiotherapy) is in lower disagreement with this factor. For factor 3, “Patient Centeredness”, there were significant differences among professional groups however these differences do not differentiate sub-groups when applied a Tukey test. There were observed higher scores in students by the courses of Physiotherapy, Gerontology, Nutrition and Nursing courses. High scores across the board showed that all students, in all professions find Patient Centeredness valuable.

DISCUSSION

Before discussing our results in the light of previous studies, it is important to note key methodological differences between our research and previous psychometric studies of RIPLS. The dimensionality of the scale in this study was evaluated using Exploratory Structural Equation Modeling ESEM(2222. Asparouhov T, Muthén B. Exploratory structural equation modeling. Struct Equ Modeling. 2009;16(4):397-438.). As the RIPLS is a Likert scale with an ordinal categorical level of measurement, the use of the polichoric correlation matrix was employed. An oblique GEOMIN rotation was then performed in the data matrix to estimate the correlations among the extracted factors(2222. Asparouhov T, Muthén B. Exploratory structural equation modeling. Struct Equ Modeling. 2009;16(4):397-438.).

The majority of previous RIPLS psychometric studies (both in the 1919. Hair JF, Anderson RE, Tathan RL, Black WC. Análise multivariada de dados. 5ª ed. Porto Alegre: Bookman; 2005. and 29 items versions) have used Principal Component analysis (PCA)(1515. Reid R, Bruce D, Allstaff K, McLernon D. Validating the Readiness for Interprofessional Learning Scale (RIPLS) in the postgraduate context: are health care professionals ready for IPL? Med Educ. 2006;40(5):415-22.,1616. El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32.,2626. McFadyen AK, Webster VS, Strachan K, Figgins E, Brown H, McKechnie J. The Readiness for Interprofessional Learning Scale: a possible more stable sub-scale model for the original version of RIPLS. J Interprof. 2005;19(6):595-603.). However, if the objective is to be able to make inferences about latent variables expressing psychological trends, PCA may not be the best approach. PCA is not a latent variable model, and as it retains items within in a specific component, thus the psychometric indexes generated by PCA, include both the common variance and the specific variance of those items. This may cause inflated saturation, which may make hamper understanding in terms of the latent variables causing items covariance. In addition, several previous studies(1515. Reid R, Bruce D, Allstaff K, McLernon D. Validating the Readiness for Interprofessional Learning Scale (RIPLS) in the postgraduate context: are health care professionals ready for IPL? Med Educ. 2006;40(5):415-22.,1616. El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32.,2626. McFadyen AK, Webster VS, Strachan K, Figgins E, Brown H, McKechnie J. The Readiness for Interprofessional Learning Scale: a possible more stable sub-scale model for the original version of RIPLS. J Interprof. 2005;19(6):595-603.) use orthogonal Varimax method as their rotation method which presumes that there is no correlation among the components/extracted factors. This method tends to over-estimate the explained variance and has a weak theoretical basis in the case of psychological constructs which are rarely divided in independent units(2727. Pasquali L. Análise fatorial para pesquisadores. Brasília: LabPam; 2005.).

Thus newer psychometric studies of the RIPLS have been performed using different data analysis methodologies taking into account those objections and constraints. A study used Exploratory Factor Analysis with oblique and orthogonal rotation, followed by SEM Confirmatory Factor Analysis(2828. Tamura Y, Seki K, Usami M, Taku S, Bontje P, Ando H, et al. Cultural adaptation and validating a Japanese version of the Readiness for Interprofessional Learning Scale (RIPLS). J Interprof Care. 2012;26(1):56-63.). Another study used the Rasch Model to evaluate the difficulty level of the RIPLS items (parameter b in the Item Response Theory – IRT)(2929. Williams B, Brown T, Boyle M. Construct validation of the readiness for interprofessional learning scale: a rasch and factor analysis. J Interprof Care 2012;26(4):326-32.). Our research methods using EFA follow the line of the more recent research studies, trying to provide validity evidence for RIPLS through analysis methodologies more adjusted to the level of measurement of the scale and to the construct to be evaluated by the instrument.

Our EFA suggested that the structural dimension of the Portuguese version of the 27-item scale could be distributed into 3 factors (Teamwork and Collaboration, Professional Identity and Patient-centredness). For the two items excluded on the factor analysis (items 18 and 20), it is worth highlighting that these items were also referred to as difficult to understand by 2 and 5 students, respectively, who took part in the pre-test interview (item 18 – I am not sure what my professional role will be; and item 20 – There is little overlap between my future role and that of other healthcare professionals).

Both previous studies that validated the expanded version of the RIPLS(1515. Reid R, Bruce D, Allstaff K, McLernon D. Validating the Readiness for Interprofessional Learning Scale (RIPLS) in the postgraduate context: are health care professionals ready for IPL? Med Educ. 2006;40(5):415-22.,1616. El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32.) also resulted in an instrument comprised of the same three factors as this present research (Teamwork and Collaboration, Professional Identity and Patient-centredness), in spite of comprising a different number of original questions. Specifically, while our scale has 27-items, Reid et al.(2525. Razali NM, Wah YB. Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. J Stat Modeling Anal. 2011;2(1):21-33.) scale has 23 items and El-Zubeir et al.(2626. McFadyen AK, Webster VS, Strachan K, Figgins E, Brown H, McKechnie J. The Readiness for Interprofessional Learning Scale: a possible more stable sub-scale model for the original version of RIPLS. J Interprof. 2005;19(6):595-603.) has 20-items. The insertion of Factor 3 – Patient centredness – expresses a change of understanding of both the education and the interprofessional practice in over three decades of debates on the constructs(1616. El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32.). This implies the repositioning of the focus of healthcare professionals from the specificity of their working areas and common beliefs to the efficacy of the technical-scientific knowledge and interventions of each profession toward the patient and his health needs, in a way to include his perception and engagement in the definition of the therapeutic plan/care plan.

The results of the subscales Cronbach’s Alpha showed an acceptable internal consistency of the scale’s Brazilian Portuguese version. The Cronbach coefficient was 0,90 and 0,75, respectively, for factor 1 Teamwork and collaboration and factor 3 Patient-centeredness. Factor 2 Professional identity presented a lower Cronbach as 0,66; however, it was also considered as adequate, as the literature considers scores higher than 0,60 as satisfactory to exploratory studies(1919. Hair JF, Anderson RE, Tathan RL, Black WC. Análise multivariada de dados. 5ª ed. Porto Alegre: Bookman; 2005.). In addition, previous validation of the RIPLS in the United Kingdom also argued that that Factor 2 (Professional Identity) appears to be less stable with items ranging in its distribution between Factor 2 and Factor 3(2626. McFadyen AK, Webster VS, Strachan K, Figgins E, Brown H, McKechnie J. The Readiness for Interprofessional Learning Scale: a possible more stable sub-scale model for the original version of RIPLS. J Interprof. 2005;19(6):595-603.). The authors suggest based on the Content Analysis, that the items of Professional Identity should be separated into two sub-scales: Negative Professional Identity and Positive Professional Identity. This may be a more useful way to construct the Brazilian version of the RIPLS because analysis identified 5 items with a negative meaning to readiness for interprofessional learning more related with competition between professionals. However, it also identified 3 items related to professional autonomy and clinical objectives that could be associated with a positive professional identity. However, the suggestion of separate factor 2 into two subscales require further investigation, as the three validation of the expanded version of the RIPLS, including this one, resulted in a three factor scale with different items composition(1515. Reid R, Bruce D, Allstaff K, McLernon D. Validating the Readiness for Interprofessional Learning Scale (RIPLS) in the postgraduate context: are health care professionals ready for IPL? Med Educ. 2006;40(5):415-22.,1616. El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32.).

The results of structural analysis on the correlation among factors showed negative association between F2 and F1 and also between F2 and F3. This result seem to provide further arguments to the interpretation that F2 Professional Identity is a representation of competitive attitudes that go against Teamwork and Collaboration, and possibly impact also the attitudes related to F3, Patient Centeredness.

The Factor “Role and responsibility” was also considered as unsatisfactory by other studies that applied the 19-item original version. Study carried out the cross-cultural adaptation of the original scale in Sweden only considered Subscale 1 – Teamwork and collaboration – which presented a Cronbach’s Alpha of 0,89. All other coefficients marked 0,48 for Subscale 2 – Professional identity – and 0,34 for Subscale 3 – Role and responsibilities(30). Other study(2626. McFadyen AK, Webster VS, Strachan K, Figgins E, Brown H, McKechnie J. The Readiness for Interprofessional Learning Scale: a possible more stable sub-scale model for the original version of RIPLS. J Interprof. 2005;19(6):595-603.) also argues that Subscale 3 – Role and responsibility – ought to be treated with some skepticism, as the original version of the RIPLS presented a Cronbach’s Alpha of 0,32(88. Parsell G, Bligh J. The development of a questionnaire to assess the readiness of health care students for interprofessional learning (RIPLS). Med Educ. 1999;33(2): 95-100.). The Roles and responsibilities factor was dropped following the trend of the two previous studies that validated the expanded version of the RIPLS(1515. Reid R, Bruce D, Allstaff K, McLernon D. Validating the Readiness for Interprofessional Learning Scale (RIPLS) in the postgraduate context: are health care professionals ready for IPL? Med Educ. 2006;40(5):415-22.,1616. El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32.) In the original version of the RIPLS the factor Role and responsibilities was composed by items 18 and 22. In this study, the item 18 was dropped out in the EFA and item 22 was found to be consistent in the Professional identity factor.

The study found that Gerontology, Nutrition and Speech Therapy students are more favorable to the interprofessional learning. On the other hand, medicine students have the lowest readiness for teamwork and interprofessional collaboration. Such results also partially corroborate other studies that validated the expanded version of the RIPLS. For example, one of that found that nursing and pharmacy undergraduate students displayed more positive attitudes regarding the collaborative teamwork than medicine students did(1616. El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32.). Similarly other study identified that postgraduate nurses and pharmacists show more interest in learning about interprofessional practice than doctors do(1515. Reid R, Bruce D, Allstaff K, McLernon D. Validating the Readiness for Interprofessional Learning Scale (RIPLS) in the postgraduate context: are health care professionals ready for IPL? Med Educ. 2006;40(5):415-22.). Research that employed the RIPLS observed that nursing and pharmacy students were more open to engage in shared learning activities whilst medicine students showed the lowest levels of interest(1111. Aziz Z, Teck LC, Yen PY. The attitudes of medical, nursing and pharmacy studentes to inter-professional learning. Soc Behav Sci. 2011;29:639-45.). Finally, another study that also applied the 19-item version of the RIPLS found that female students, as well as nursing students, showed more positive attitudes toward shared interprofessional learning than medical student(1212. Wilhelmsson M, Ponzer S, Dahlgren LO, Timpka T, Faresjö T. Are female studentes in general and nursing students more ready for teamwork and interprofessional collaboration in healthcare? BMC Med Educ. 2011;11:15. doi 10.1186/1472-6920-11-15). Based on these results, the last one suggests that the IPE-oriented activities need to promote the engagement of medical students.

CONCLUSION

In conclusion the RIPLS version in Brazilian Portuguese has good internal consistency and reliability. The validated version of RIPLS consists of an efficient assessment tool for IPE initiatives in Brazil. In this sense, this study contributes toward the intensification of the IPE and the integrated, collaborative teamwork in the services that comprise the Brazilian Unified Health System (SUS), as well as encourages its use in cross-cultural comparative studies. As commented on in other psychometric studies of the RIPLS scale, the factor of Professional identity is the least stable, and requires further exploration in terms of developing the items.

Teamwork is listed among the operational guidelines for restructuring the Unified Health System in Brazil. However, it has been stated that team-based care has advanced more as a proposition than as an action truly translated into practice among other reasons because of the lack of widespread IPE and research about it. Therefore a tool such as RIPLS and its Portuguese version can be helpful for Brazilian education planners and policy makers to design better IPE training programs investigate healthcare students’ attitudes related to shared learning experiences and their relationship with professional practice in SUS services.

Acknowledgements

We are grateful to the authors of the original scale in its expanded version, Professors John Bligh and Karen Mattick, that authorized its validation in Brazil and made available the original version, in order to allow the development of this study.

REFERÊNCIAS

  • 1
    Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes (update). Cochrane Database Syst Rev. 2013;(3):CD002213.
  • 2
    World Health Organization. Framework for action on interprofessional education & collaborative practice. Geneva: WHO; 2010.
  • 3
    Batista NA. Educação interprofissional em saúde: concepções e práticas. Cad FNEPAS. 2012;(2):25-8.
  • 4
    Peduzzi M, Norman I, Germani ACGG, Silva JAM, Souza GC. Interprofessional education: training for healthcare professionals for teamwork focusing on users. Rev Esc Enferm USP [Internet]. 2013 [cited 2014 Oct 10];47(4):977-83. Available from: http://www.scielo.br/pdf/reeusp/v47n4/en_0080-6234-reeusp-47-4-0977.pdf
    » http://www.scielo.br/pdf/reeusp/v47n4/en_0080-6234-reeusp-47-4-0977.pdf
  • 5
    Feuerwerker LCM, Sena RR. Contribuição ao movimento de mudança na formação profissional em saúde: uma avaliação das experiências UNI. Interface Comun Saúde Educ. 2002;6(10):37-50.
  • 6
    Barreto ICHC, Loiola FA, Andreade LOM, Moreira AEMM, Cavalcanti CGCS, Arruda CAM, et al. Development of interprofessional collaborative practices within undergraduate programs on healthcare: case study on the Family Health Alliance in Fortaleza (Ceará, Brazil). Interface Comum Saúde Educ. 2011;15(36):199-212.
  • 7
    Brasil. Ministério da Educação. Instituições de Educação Superior e Cursos Cadastrados [Internet]. Brasília; 2012-2013 [citado 18 fev. 2013]. Disponível em: http://emec.mec.gov.br/
    » http://emec.mec.gov.br/
  • 8
    Parsell G, Bligh J. The development of a questionnaire to assess the readiness of health care students for interprofessional learning (RIPLS). Med Educ. 1999;33(2): 95-100.
  • 9
    Thannhauser J, Russel-Mayhew S, Scott C. Measures of interprofessional education and collaboration. J Interprof Care. 2010;24(4):336-49.
  • 10
    Coster S, Norman I, Murrells T, Kitchen S, Meerabeau E, Sooboodoo E, et al. Interprofessional attitudes among undergraduate students in the health professions: a longitudinal questionnaire survey. Int J Nurs Stud. 2008;45(11):1667-81.
  • 11
    Aziz Z, Teck LC, Yen PY. The attitudes of medical, nursing and pharmacy studentes to inter-professional learning. Soc Behav Sci. 2011;29:639-45.
  • 12
    Wilhelmsson M, Ponzer S, Dahlgren LO, Timpka T, Faresjö T. Are female studentes in general and nursing students more ready for teamwork and interprofessional collaboration in healthcare? BMC Med Educ. 2011;11:15. doi 10.1186/1472-6920-11-15
  • 13
    Lie DA, Fung CC, Trial J, Lohenry K. A comparison of two scales for assessing health professional student’s attitude toward interprofessional learning. Med Educ Online 2013;18:21885. doi: 10.3402/meo.v18i0.21885.
  • 14
    Mattick K, Bligh J. Readinss for interprofessional learning scale. In: Bluteau J, editor. Interprofessional education. London: MacMillan; 2007.
  • 15
    Reid R, Bruce D, Allstaff K, McLernon D. Validating the Readiness for Interprofessional Learning Scale (RIPLS) in the postgraduate context: are health care professionals ready for IPL? Med Educ. 2006;40(5):415-22.
  • 16
    El-Zubeir M, Rizk DE, Al-Khalil RK. Are senior UAE medical and nursing students read for interprofessional learning? Validating the RIPL scale in a Middle Eastern context. J Interprof Care. 2006;20(6):619-32.
  • 17
    Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25(24):3186-91.
  • 18
    Wynd CA, Schmidt B, Schaefer MA. Two quantitative approaches for estimating content validity. West J Nurs Res. 2003;25(5):508-18.
  • 19
    Hair JF, Anderson RE, Tathan RL, Black WC. Análise multivariada de dados. 5ª ed. Porto Alegre: Bookman; 2005.
  • 20
    Horn JL. A rationale and technique for estimating the number of factors in factor analysis. Psychometrika. 1965;30:179-85.
  • 21
    Venables WN, Smith DM. The R Development Core Team. An introduction to R, Version 2.2.0 R-Project [Internet]. [cited 2013 Oct 10]. Available from: http://CRAN.R-project.org
    » http://CRAN.R-project.org
  • 22
    Asparouhov T, Muthén B. Exploratory structural equation modeling. Struct Equ Modeling. 2009;16(4):397-438.
  • 23
    Muthén LK, Muthén BO. Mplus user’s guide. 2nd ed. Los Angeles: Muthén & Muthén; 2001.
  • 24
    Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. 7nd ed. New Jersey: Pearson Education International; 2010.
  • 25
    Razali NM, Wah YB. Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. J Stat Modeling Anal. 2011;2(1):21-33.
  • 26
    McFadyen AK, Webster VS, Strachan K, Figgins E, Brown H, McKechnie J. The Readiness for Interprofessional Learning Scale: a possible more stable sub-scale model for the original version of RIPLS. J Interprof. 2005;19(6):595-603.
  • 27
    Pasquali L. Análise fatorial para pesquisadores. Brasília: LabPam; 2005.
  • 28
    Tamura Y, Seki K, Usami M, Taku S, Bontje P, Ando H, et al. Cultural adaptation and validating a Japanese version of the Readiness for Interprofessional Learning Scale (RIPLS). J Interprof Care. 2012;26(1):56-63.
  • 29
    Williams B, Brown T, Boyle M. Construct validation of the readiness for interprofessional learning scale: a rasch and factor analysis. J Interprof Care 2012;26(4):326-32.
  • 30
    Lauffs M, Ponzer S, Saboonchi F, Lonka K, Hylin U, Mattiasson AC. Cross-cultural adaptation of the Swedish version of Readiness for Interprofessional Learning Scale (RIPLS). Med Educ. 2008;42(4):405-11.
  • Financial support: Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP). Processos 2010/03107-2 e 2011/11772-9. Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq). Processo 200534/2010-6.

Publication Dates

  • Publication in this collection
    Dec 2015

History

  • Received
    30 Nov 2014
  • Accepted
    10 Feb 2015
Universidade de São Paulo, Escola de Enfermagem Av. Dr. Enéas de Carvalho Aguiar, 419 , 05403-000 São Paulo - SP/ Brasil, Tel./Fax: (55 11) 3061-7553, - São Paulo - SP - Brazil
E-mail: reeusp@usp.br