Cross-cultural adaptation of the EFFECT questionnaire into Brazilian Portuguese

Introduction: Clinical teaching is based on a real work environment, in professional practice settings, such as health services and units, under the supervision of the preceptor. Providing medical teachers with an assessment of their teaching skills is a powerful tool for improving clinical learning for students in training. In this context, the EFFECT (Evaluation and Feedback for Effective Clinical Teaching) questionnaire was developed by Dutch researchers in 2012 for teacher evaluation, being validated based on the literature about medical teaching in the workplace and incorporates the skills of the Canadian competency-based medical curriculum. Objective: To translate and cross-culturally adapt into Brazilian Portuguese and to validate the EFFECT questionnaire for teacher evaluation by Medical students. Method: Cross-cultural adaptation with the following steps: initial translation of the English version, synthesis of translated versions, back-translation, creation of a consensual version in Brazilian Portuguese, with adaptation, review, and analysis of content validity by an expert committee, pre-test with retrospective clarification interview, and reliability analysis by factorial analysis and internal consistency test (Cronbach’s alpha coefficient). Result: In the translation and back-translation stages, the disagreements were related to the use of synonyms and none of the items were modified in terms of their understanding, but in terms of adaptation into the Brazilian context. The evaluation of the expert committee showed the versions maintained the semantic and idiomatic equivalences of the content. Eighty-nine students participated in the pre-test. The internal consistency of the EFFECT questionnaire in Brazilian Portuguese was excellent for all domains, with Cronbach’s alpha coefficient ranging from 0.82 to 0.94. Conclusion: The translated and adapted version of the EFFECT questionnaire into Brazilian Portuguese is equivalent to the original instrument and has evidence of high validity and reliability, being able to constitute a national tool to evaluate the efficiency of clinical medicine teaching.


INTRODUCTION
The several changes that have occurred in the field of medical education in the last decades, towards the construction of competency-based curricula (CBC) and evaluation of results and performance 1,2 , have led to several intervention proposals in the training of health professionals, generating a new social mission of the Higher Education Institutions (HEIs) 3 . Competency matrices for medical education, both for undergraduate courses, as proposed by the AAMC (Association of American Medical Colleges), and for medical residency, such as CanMEDS, of the Canadian Royal College of Physicians and Surgeons, and the Milestones Project, of the North American Accreditation Council for Graduate Medical Education, constitute the training base for the majority of young medical apprentices in the Western world and are examples for medical schools worldwide [4][5][6][7][8] .
In Brazil, the National Curriculum Guidelines (NCGs) propose the centralization of the teaching of medical training in primary health care (PHC) 9,10 . Therefore, there is a growing interest in the search for innovative ways of teaching that involve ethical, critical, reflective and transformative training 10 and that attract students to PHC and public health 8,11 . The creation of integrated curricula that demonstrate the early integration of students since the first years of undergraduate school with professionals and users of health services, in longitudinal clinical practices is recommended 12-14. In parallel with the development of concepts related to teaching over the years, in Brazil, national systems for institutional assessment of higher education have been created and improved, including accreditation processes 15 . Usually, the assessment includes aspects related to the agreement with the NCGs, among others: teaching, research, extension, social accountability, facilities, student performance and the faculty 16 .
The teaching-service integration indicated by the NCGs may encounter obstacles, such as the presence of preceptors and teachers with insufficient training for teaching 14 . The teacher plays a leading role in curricular changes 17 and, for their effective implementation, it is necessary to develop instruments to evaluate the teaching process and for the global assessment of programs and curricula 18 .
Most of the research on medical education assessment focuses on the use of several instruments, based on the competence view that consists of knowledge, skills, and attitudes, in a disintegrated way [19][20][21] . As for teacher evaluation, there is no standardization, and most initiatives are based on assessments made by the students, with excessively broad, subjective criteria, often answered based on the teachers' personality and not on their skills, and subject to confounding factors [22][23][24][25] .
An evaluation method for teachers requires some characteristics to be effective: to be continuous and systematic; include all actors involved in the teaching-learning process; generate information about the educational diagnosis; lead to reflections that can contribute to the improvement of the course and teacher development. It is also necessary to safeguard the anonymity of students and ensure that teachers are not present during the assessment 26 .
Most medical professors are trained in a hospital-centered model, in which most of their experiences derive more from their clinical knowledge than from specific teaching training.
Therefore, teacher training is essential for the development of competency-based education, as it is undeniable that isolated clinical training does not provide adequate preparation for the development of health professionals into teachers 17 .
Understanding the importance of teacher evaluation, especially clinical preceptors, and considering the scarcity of instruments for practice evaluation and the satisfactory training of medical competencies, a group of Dutch researchers developed an instrument for teacher assessment by residents called the "Evaluation and Feedback for Effective Clinical Teaching" (EFFECT Questionnaire), which was validated based on the literature on medical teaching in the workplace and incorporating the CanMEDS competencies 27,28 .
For the best student performance and training of new doctors, one can realize how crucial the continuous processes of faculty and clinical preceptor training are, and the greater application of the tools that assist this process. Therefore, it is undeniable the relevance of the translation and adaptation to the Brazilian reality of an assessment instrument capable of broadly evaluating all aspects inherent to teaching practice, aiming at encouraging continuous evaluation among students, teachers, and the Educational Institution, as well as the promotion of theoretical-practical integration and teaching-society.
This article aims to present the process of translation and cross-cultural adaptation of the EFFECT questionnaire into Brazilian Portuguese, which includes testing its validity and reliability so that it can be used for teaching evaluation by students in Brazilian medical schools.

Study design
Translation and cross-cultural adaptation that included the following phases 29

Participants
The study sample comprised 89 medical students from UNIFOR, aged at least 18 years, who agreed to participate in the study and had available time and Internet access. The sample recruitment to participate in the research was carried out by public call, with wide dissemination of the research, in physical form through posters in the educational institution and virtually via the university's own communication application. The questionnaire was sent online, along with the free and informed consent form for obtaining the answers online, using the Google Forms® platform (https://www.google.com/forms/about/).

Procedures
First stage: two certified bilingual translators, independently carried out the initial translation of the English version into Brazilian Portuguese, from June to July 2019.
Only one of the translators was aware of the concepts the questionnaire intends to measure, to provide a translation that most closely resembles the instrument in the English version.
There was a third translation by one of the study's authors, fluent in English. Therefore, the translated versions T1, T2 and T3 were generated. (absolutely equivalent item). The items that received a score of 1 or 2 were reviewed. The acceptable level of agreement of the content among the members of the expert committee must be at least 0.80 and, preferably, higher than 0.90 29,34 . The   committee was responsible for a review of all the translation   phases and evaluated whether the translated versions (T1, T2,  T3, T12,
Since this article has two related but different objectives, the results will be presented in 2 separate sections. The first one will deal with the results of the process of translation and cross-cultural adaptation of the questionnaire; and the second will show the data pertinent to the description of the characteristics of variables and the internal consistency psychometric data.

Translation and cross-cultural adaptation of the EFFECT questionnaire content
During the initial translation process from the English version into Brazilian Portuguese, some disagreements can be observed between the translators (T1 and T2) regarding the two translations. The differences, however, were minimal, using synonymous words and writing forms that did not change the meaning; for instance: Translator 1 translated the phrase "treat patients respectfully" as "tratar os(as) pacientes de maneira respeitosa", while translator 2 translated as "tratar o paciente de forma respeitosa". The most critical items were reviewed in the following stages of analysis by the expert committee and after interviewing the pre-test respondents.
In the back-translation stage, there were no difficulties when back-translating the items. Although the BT1 and BT2 versions did not have many items that were literally equal to the English version, they maintained the semantic equivalence. Throughout the statements, the words "instructor/ supervisor" were translated as "preceptor/supervisor". In item 1, originally "ask for a patient's history", we chose the literal translation "perguntar o histórico do(a) paciente" instead of the expressions "realizar anamnese" or "coletar história", which are common in Brazil, to maintain better translation equivalence.
The expression "have a bad news conversation" in item 11 was However, after responses and interviews with the respondent students in the pre-test, it was observed that the word "portfolio" was unknown for some of them, probably because it is a non-standardized evaluation method for all courses or HEIs. Therefore, a short explanatory text was added after the item. In the "avaliação" (Evaluation) statement section, it was necessary to add the words "formativa" and "somativa" to facilitate the understanding of the sentence: "Durante seu treinamento, ocorreram várias entrevistas sobre o seu progresso. Pode-se enfatizar -quando necessário -os ajustes (durante uma revisão de progresso -formativa), ou na conclusão de parte da sua formação (durante uma avaliação de rodízio ou entrevista de avaliação -somativa)".
On multiple occasions, English verbs followed by the pronoun "me" were inverted; for instance "gives me" was translated into "me dá", as it is the more popular idiomatic expression in the spoken Brazilian Portuguese language.
The word "feedback", which can be translated as "retroalimentação" or "retorno", remained in English due to its wide use in the educational context and because its use is disseminated in the Brazilian Portuguese language.

Description of the variables and analysis of construct validity and internal consistency of the questions
The study of the characteristics of the questions shows averages ranging from 4.96 (item 28 -"retorna ao feedback dado anteriormente") to 5.81 (item 43 -"me trata respeitosamente").
The factorial loads ranged from 0.2816 (item 3 -"realizar abordagens clínicas") to 0.9109 (item 33 -"como explico minhas escolhas por uma abordagem particular") ( Table 2). The factorial load can be defined as the correlation of the variable (item) with the factor, indicating how much each variable contributes to the factor (questionnaire construct). If this load has a positive value, it means that the variable is positively correlated with the factor, and, if it has a negative value, this correlation is negative.
They are considered significant when they exceed the absolute value of 0.30 44 . Thus, it is observed that, except for item 3, all items of the translated questionnaire were positively correlated to the analyzed domains.
The description of the data according to the domains is shown in Table 3. The domain with the lowest average was "fornecendo feedback" (average 5.19; SD 0.857) and the one with the highest average was "características gerais do preceptor/ supervisor" (mean 5.57; SD 0.714). The lowest internal consistency was found for the domain "cuidando do trabalho significativo" (0.83), and this value being classified as acceptable 36 . Abbreviations: a: SD = standard deviation; b: NAE ''not-yet able to evaluate'' or "ainda não é capaz de avaliar".

DISCUSSION
Over the past few decades, profound changes have occurred in the medical education scenario, such as the introduction of new teaching and assessment techniques, as well as changes in the profile and involvement of university students 45 . These belong to "Generation Z" and grew up immersed in this reality of technological innovations 46,47 .
For this generation, it is necessary to use student-centered methodologies, favoring collaboration, interdisciplinarity, autonomy and critical and citizen education, supporting To follow these changes, it is necessary to review the teacher's role, abandoning the concept of academic or encyclopedic professor, as a mere transmitter of knowledge, and that of the technical specialist 48 , and to start exercising a variety of duties 37 . Therefore, the HEIs have emphasized the evaluation of their teachers based on their clinical competency, teaching skills, personal qualities, involvement with students, quality in providing care to patients, in addition to the ability to provide feedback and guidance throughout the teaching activities 49 .
In this study, the translation, cross-cultural adaptation and evaluation of the validity and reliability of the EFFECT questionnaire were carried out in Brazilian Portuguese, demonstrating that the instrument showed to be valid and reliable to be used in the context of Brazilian medical education.
The results show that there were no major difficulties during the stages of translation, back-translation, adaptation and validation by the experts, considering that many educational concepts and values are common to several medical schools worldwide. It is important to note that the respondents' difficulties and doubts in relation to the questionnaire were observed and considered for the preparation of the final version.
The results of this study indicate that the Brazilian version of the EFFECT questionnaire has acceptable psychometric properties and can be used to evaluate the teachers' clinical teaching from the perspective of undergraduate medical students. All domains showed a satisfactory reliability coefficient. It is noteworthy that the participating students indicated that some items 'not-yet able to evaluate' (NAE). Most of these items are due to the domain "avaliação", even though it was answered only by those who responded affirmatively to the domain's introductory question ("Seu preceptor/supervisor conduz revisões de progresso ou entrevistas de avaliação com você?"). This fact may be associated with a non-standardization of the evaluation processes of the students' competencies or the fact that these are evaluated in other ways, not necessarily the one present in the questionnaire. Only 4 items of a total of 55 in the questionnaire had an average score lower than 5 (answer equivalent to "Bom, continue assim", demonstrating a good evaluation of the faculty by the students. When the domains were analyzed, all of them obtained an average greater than 5. In the domain "o(a) preceptor(a) como modelo", items 6 ("aplicar resultados de pesquisas acadêmicas") and 14 ("pede a opinião de outros sobre seu próprio desempenho") were the only ones with an average lower than 5. As for the domain "planejamento de atividades de treinamento", a high score (average of 5.52) was obtained when questioning the teacher's availability when the student deemed it necessary. However, there was a decrease in the score (5.38) when asked whether the preceptor sets aside time for the student when they need it, and a greater decrease (5.15) when asked whether the teacher reserves time to supervise/advise the students. Regarding "fornecendo feedback", two items had an average lower than 5: "me deixa pensar sobre meus pontos fortes e fracos" and "retoma feedbacks dados previamente".
These results demonstrate how important and appreciated by the students is the fact that teachers are open to welcoming their needs and that they plan specific moments for evaluation. The feedback strategy provides the student with a self-assessment and should have an impact on their development. These are complex and dynamic moments of student-teacher interaction 50 . An effective feedback should be assertive, respectful, descriptive, timely and specific 51 . It is expected that the teacher be able to mediate, listen and dialogue, favoring the exchange of ideas and the articulation between the knowledges 52 .
It can be observed that in the "habilidades de ensino" It is known that there is a lack of teaching professionalization among medical professors, mainly related to teaching skills, and often associated with a lack of knowledge of the curricular structure and the recently-graduated student profile desired by educational institutions. It is necessary to overcome information overload-based teaching, where the teacher is focused on the technique and the mere transmission of information, without integrating disciplines and using student assessment methods, with a sometimes punitive character. This situation can be an obstacle for doctors who do not consider teaching as their main professional activity 54 .
University-level education can be understood in two ways: non-professional, which considers the act of teaching as repeating previous models based on everyday practical experience; and the professional one, which defends teaching as a complex act consisting of skills that can be acquired, improved and expanded through a consistent training process 55 . The criterion of being considered a good medical professional in a specific area of clinical practice, although common when hiring teachers for medical courses, is not enough to guarantee quality teaching.
Regarding the domain "características gerais do(a) preceptor(a)", the item with the highest average (5.81) was related to a respectful student-teacher relationship (item 43). The lower averages were those for items 45 ("deixa claro que posso contar com ele(a)'') with an average of 5.57, and 47 ("está aberto(a) a questões/problemas pessoais") with an average of 5.24. Although traditionally, the actions of guiding, orienting and advising in achieving personal goals, also seeking interpersonal and psychosocial development beyond the professional environment are considered the "Mentor's role, and not specifically the "Preceptor's" or the "Supervisor's", roles that are more associated to teaching in real clinical situations 56 , these results demonstrate that there is still room for improving the student-teacher relationship.
The difficulty in obtaining the adherence of medical teachers to curricular reforms has been recurrent and can be explained by several factors, such as the non-professionalization of the teaching function, partial dedication to schools and insecurity. A study sought to analyze how the medical teachers' characteristics at a university in Minas Gerais had an impact on the way students evaluate teaching. The data showed that teachers with less time since graduation, less time working as a teacher and less experience but are motivated by the pleasure of teaching, are better evaluated than older and more experienced ones 52 .
In Brazil, a cross-sectional study was carried out with 28 students on the competencies of teachers of the medical course at Universidade Federal do Amapá, using focal groups. The students emphasized the importance of providing formative feedback to teachers. For them, the evaluation helps so that the teacher identifies their deficiencies and can seek ways to solve them, in addition to increasing the bond with the students. The study also showed, in the students' opinion, that the best way to assess teachers would be through a structured questionnaire with objective and subjective items 26 .
Any instrument designed to assess the quality of education must have good validity and reliability. Moreover, the generated results need to be useful to provide improved teaching, support and motivation for the teachers, academic advancement and promotion, curricular evolution, in addition to providing subsidies for the creation of faculty development programs 57 . These characteristics are present in the EFFECT questionnaire.
In comparison with the original study for the development of the EFFECT 27 questionnaire and another previous study about its validation in Lithuanian 33 , there are some differences in our research that may have influenced the results. The present study was carried out with a smaller number of participants. Moreover, previous studies used the EFFECT questionnaire with resident physicians, while our study applied it to undergraduate medical students. Another limitation of the present study is that there was no re-test.
The EFFECT questionnaire in its Brazilian Portuguese version showed to be effective and easy to apply, generating information that can be useful for educational institutions to prepare faculty development plans. The university where this study was developed has a Professional Development Program in Education (PDPE, Programa de Desenvolvimento Profissional em Educação), coordinated in partnership with the Center Pedagogical Advisors and the Educational Technologies Center, where courses are systematically offered to all university teachers. The PDPE proposes continuing education on current and innovative topics in the context of higher education for teachers. Therefore, there is an expectation that the instrument will be adopted to generate information for the process of planning institutional actions for continuing teacher education.

CONCLUSIONS
The results of this study indicate that the Brazilian Portuguese version of the EFFECT questionnaire has acceptable psychometric properties for the evaluation of clinical teachers by students from medical courses at Brazilian Higher Education Institutions. It is noteworthy that there is no available validated instrument yet in the local literature that can measure the same teaching competencies as the EFFECT.

ACKNOWLEDGMENTS
We would like to thank Lia Fluit, associate professor at Hoofd

CONFLICTS OF INTERESTS
The authors declare no conflicts of interest.

FUNDING SOURCES
The authors declare no funding sources.