Validity and Reliability of the Portuguese-Language Version of the Maastricht Clinical Teaching Questionnaire

ABSTRACT

Introduction

In clinical learning scenarios, preceptors should mediate between the knowledge students already possess and the knowledge they need to acquire, integrating theory and practice into the patient care context. In order to make this possible, we must train preceptors and develop instruments for evaluating their performance as educators. In 2008, the Maastricht Clinical Teaching Questionnaire (MCTQ) was developed in the Netherlands with the purpose of enabling students to evaluate their preceptors, in order to provide preceptors with feedback regarding their teaching skills and the supervision provided by them during the internship period. Validating an instrument that has been translated is important and valuable, because it enables certain phenomena to be evaluated and investigated in different countries.

Objective

To validate the Portuguese-language version of the MCTQ.

Methodology

This is an instrument validation study, with research carried out in four different hospitals in the Northeast Region of Brazil. It applied the Portuguese-language version of the MCTQ to a non-probability sample composed of 246 fifth- and sixth-year students of Medicine from four different institutions in the state of Pernambuco. Kappa values were used to evaluate the validity of the construct, and reliability was measured using standardized Cronbach’s alpha coefficient. The reproducibility level of the MCTQ was calculated using the Student’s t-test for repeated measures, comparing test and re-test values. This study was approved by the Human Research Ethics Committee of the Faculdade Pernambucana de Saúde.

Results

The kappa index was between 0.527 and 0.71, and Cronbach’s alpha coefficient was between 0.77 and 0.954, demonstrating that the instrument has good levels of agreement and internal consistency, respectively. With respect to reproducibility, all coefficient correlation values found were significant and showed good magnitude (≥0.72).

Conclusion

The Portuguese-language version of the MCTQ proved to be reliable and valid for use in Portuguese-language settings, and can be useful as an instrument for promoting pedagogical improvements in undergraduate and graduate courses, especially in health sciences.

Validation study; Educational evaluation; Preceptorship; Medical education

RESUMO

Introdução

Num cenário de aprendizagem clínica, preceptores devem se comportar como mediadores entre os saberes que o estudante já tem e os que necessita adquirir, integrando a teoria e a prática no contexto da assistência ao paciente. Para isso, é necessário capacitar preceptores e desenvolver instrumentos capazes de aferir o desempenho de tais docentes. Em 2008, foi desenvolvido na Holanda o Maastricht Clinical Teaching Questionnaire (MCTQ), destinado à avaliação de preceptores pelos estudantes com o objetivo de proporcionar feedback a esses sobre suas habilidades de ensino em relação à supervisão realizada durante o estágio. A validação de um instrumento para outro idioma é de grande valia, uma vez que permite avaliar e investigar determinado fenômeno em diferentes países.

Objetivo

Realizar a validação do MCTQ para a língua portuguesa.

Metodologia

Trata-se de um estudo de validação de instrumento de pesquisa realizado em quatro hospitais do Nordeste brasileiro com a aplicação da versão adaptada para a língua portuguesa do MCTQ em uma amostra não probabilística composta por 246 estudantes de Medicina do quinto e sexto anos de quatro diferentes instituições de ensino do Estado de Pernambuco. Para avaliar a validade do construto, foi utilizado o índice Kappa. A confiabilidade foi medida por meio do Coeficiente Alfa de Cronbach padronizado. O nível de reprodutibilidade do MCTQ foi calculado pelo teste t de Student para medidas repetidas, comparando valores do teste e do reteste. A pesquisa foi aprovada no Comitê de Ética e Pesquisa da Faculdade Pernambucana de Saúde.

Resultados

O índice Kappa variou entre 0,527 e 0,710, e o Coeficiente Alfa de Cronbach de 0,77 a 0,954, comprovando bom grau de concordância e de consistência interna do instrumento, respectivamente. Em relação à reprodutibilidade, todos os valores de coeficiente de correlação encontrados foram significativos e de boa magnitude (≥ 0,72).

Conclusão

A versão em português do MCTQ mostrou-se confiável e válida para uso na língua portuguesa e pode ser útil como instrumento a promover melhorias pedagógicas nos cursos de graduação e pós-graduação, especialmente aqueles relacionados às ciências da saúde.

Estudos de Validação; Avaliação Educacional; Preceptoria; Educação Médica

INTRODUCTION

Throughout history, the teaching of medicine has involved transmitting knowledge from a more experienced doctor to younger apprentices, without there being specific pedagogical training for this teaching-learning process. Teaching was based on the younger doctors directly following and observing the procedures performed by the more knowledgeable doctors. ¹

Even today, medical students do not receive adequate preparation for teaching. The pre-requisite for teaching, more often than not, is experience in a specific medical discipline and teaching, in turn, is often practiced in a traditional format, in which the student is a mere spectator and the teacher the holder and transmitter of knowledge. ^2,3

At the beginning of the 20^thcentury, the classic study known as the Flexner report was published. ⁴ Highlighted among the proposals it contained was having an initial cycle or period for basic disciplines clearly separated from another dedicated to clinical studies. However, this format has been widely criticized, even though it is often used in our universities. ⁵ On the other hand, Flexner ⁶ did support practical activities, both in the laboratory and in the clinic. In his view, the student only learned by doing. Hence, he was against teaching based purely on lectures and the prioritizing of memorizing as a learning method.

From this perspective, and in accordance with the National Curricular Directives for Medical Degree Courses, clinical students are subjected to teaching methodologies that favor the active participation of the student in different scenarios, particularly in health units belonging to the Unified Health System (SUS – Sistema Único de Saúde), attending at different levels, providing the student with knowledge and experience, in a practical and increasingly complex manner, in situations inherent to the profession. ⁷

In the process of implementing the Brazilian government’s “More Doctors Program” (Programa Mais Médicos), the Ministries of Health and of Education have encouraged the creation of new medical schools throughout the country. Associated with this growth is the requirement that the new professionals trained have a more critical, ethical and reflective profile, directed particularly at the requirements of SUS. ⁸

In this new context, it is insufficient for the student to acquire cognitive knowledge and demonstrate technical abilities. ⁹ It is now required that the trained professional also has the attitudes, values and behavior expected of a doctor. The doctor must “think, act and feel like a doctor”. ¹⁰

Hence, in parallel with the aims of this concept of this new type of professional, it has become urgently necessary that preceptors are adequately prepared for the changes underway. A good preceptor needs to be aware not only of how to act as a facilitator for the technical learning in the specific medical discipline, but must also serve as a model in transmitting moral values by their attitudes and behavior with patients. From the perspective of the students, the model preceptor also shows enthusiasm for teaching, is encouraging and accessible, appears interested in the development of their students and creates a positive learning environment, among many other qualities.^11,12.

In addition to the importance given to the requirement that preceptors are qualified, it is necessary to have instruments capable of evaluating their performance. For this purpose, the feedback provided by students can be used as relevant data to help the teachers to improve their teaching skills. ¹³

In this regard, there are various instruments that aim to provide data about the quality of the teaching by preceptors. These are questionnaires in which the medical students and resident doctors respond to questions that evaluate their teachers. Among these, the Maastricht Clinical Teaching Questionnaire (MCTQ) was developed in 2008 to enable students to evaluate preceptors with the intention of providing feedback to them about their teaching skills during the internship stage of the course. The theoretical foundation of the MCTQ is cognitive learning and its product was obtained from the extensive description of this model. ¹⁴

In 2010, Stalmeijer and collaborators ¹⁵ demonstrated the validity of the MCTQ for use in clinical internships. Accordingly, the MCTQ can be considered to be not only a valuable instrument for evaluation and feedback, but a tool that enables self-reflection by preceptors on pedagogical development, thereby promoting improvements in clinical teaching.

So far, there has been no questionnaire available in the Portuguese language, which has been duly validated for use in our country, with the characteristics described above. However, in 2015 a study was performed that provided a Portuguese translation and adaptation of the MCTQ for use in Brazil, ensuring the validity of the content. It is the first instrument that evaluates clinical skills to be adapted transculturally to the Portuguese language. The national version of the MCTQ was found to be easily understood by the students and was well accepted in a pilot test. ¹⁶

Nevertheless, to be applied in practice to a specific population, it is important that the instrument meets the requirements of validity and reliability. Validity refers to the degree to which an instrument truly measures the variable it is intended to measure, that is, measures what it is supposed to measure. Reliability is a measure of the confidence that it inspires for providing coherent and repeatable results. ^17,18

The transcultural adaptation and validation of an instrument to another language is of great value since it enables evaluation and research of specific phenomena in different contexts and can be used not only in local studies, but also in international and multicentric ones, which occur increasingly frequently in the scientific community. This enables comparisons to be made between distinct populations and takes less time than formulating a new instrument. ¹⁹ In this context, considering the lack of a Brazilian instrument with previously tested validity and reliability to evaluate clinical teaching, the objective of this study is to validate the MCTQ for the Portuguese language.

METHODS

About the instrument

The MCTQ covers five domains: (1) modeling; (2) coaching; (3) articulation; (4) exploration and (5) safe learning environment, involving fifteen items to evaluate the performance of a preceptor and by which the student indicates their agreement on five-point Likert scale (1 = fully disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = fully agree). The modeling domain includes aspects related to the preceptor demonstrating a task in practice and the observation of this by the student, as well as the role of modeling for health professionals. The three items making up the coaching domain reflect the feedback provided by the preceptor after the student has executed a task, the adjustment of the activities to the student’s level of experience, and the opportunity to perform them independently. The articulation domain consists of three items relating to the preceptor encouraging the student to reason, by asking for explanations for their actions, and the questioning and exploration of their strong and weak points. The two items making up the exploration domain consist of the encouragement to the student for preparing for and achieving the learning goals. Finally, the last domain is related to the safe learning environment. Its three items address aspects related to the learning environment, the preceptor’s respect for the student, and their interest in the student’s learning. ¹⁴

Translation and transcultural adaptation

The phases of the transcultural adaptation of the MCTQ were achieved in accordance with the recommendations in the internationally accepted literature. ²⁰ The five phases implemented in the process consisted of: 1) direct translation; 2) combination of translations; 3) reverse translation; 4) consolidation by a committee of judges; 5) pre-test (viability and applicability). ^21,22

Step 1 involved two independent, qualified bilingual translators, who are native-speakers of Portuguese as spoken in Brazil, the target language to which the questionnaire was translated, from English, the original language. In step 2, the translations were compared by the two translators and researchers and the differences between the translated versions were identified, together with the modifications required to achieve consensus, giving rise to a combined version. During step 3, a reverse translation of this version was performed by two bilingual translators with the same mother tongue as the original questionnaire. Accordingly, after this step the Portuguese translation of the instrument for the purpose of generating a pre-final version had been produced.

For step 4, a committee was formed of specialists with significant practical experience of the area in question and representatives of the target population. This committee of judges consisted of 10 specialists on the theme: 04 specialists in education, 01 specialist in research into transcultural adaptation and validation, 02 medical students who had completed 80% or more of the clinical rotation and 03 clinical preceptors. This group met with the responsible researchers to carefully analyze the translated versions and the pre-final version with the objective of facilitating the analysis and obtaining the final version used for the validation. At the end of the transcultural adaptation process, step 5 was applied to the version of the MCTQ adapted to the Portuguese language in a sample of students with similar characteristics to the population of interest. There was good acceptance during the pre-test and the students stated that the questionnaire was clear, easy to understand and complete, and had instructions that were quick and simple to follow. ¹⁶

Study population and data collection

The sample was non-probabilistic, consisting of 246 students of both sexes, recruited voluntarily, studying the final two years of the Medical course at four higher education institutions in the state of Pernambuco who were conducting compulsory internships (placements) in four hospitals in the Northeast of Brazil, connected to the SUS, during the period of the research.

Of the four units, three represented the largest health services located in the São Francisco Valley, a region that encompasses the cities of Petrolina – State of Pernambuco (PE) – and Juazeiro – State of Bahia (BA). The fourth hospital is located in the city of Recife, capital of the State of Pernambuco. The version of the MCTQ adapted to the Portuguese language was applied as the data collection instrument. Using the questionnaire, the students evaluated the preceptor at two distinct times with a minimum interval between them of one week (test and re-test). Unlike the original study in which the questionnaire was completed by all participants anonymously, in this it was the choice of the student whether to be identified or to be anonymous.

The questionnaire was applied in the period between March and August 2016 by a team consisting of three professionals trained in applying the MCTQ. They had no hierarchical relationship with the participating students and, at no time, were the authors evaluated, thereby avoiding any conflict of interests. Note, also, that the preceptors or supervisors evaluated did not have access to the questionnaire.

The research was performed in accordance with Resolution 466/12 of the National Health Council and was approved by the Ethics Committee for Research on Humans of the Pernambucana Health Faculty under CAEE 51892515.7.1001.5569.

Statistical analysis

The data obtained was typed twice into an Excel® (Microsoft Corporation, Redmond, WA, United States, Release 14.0.7173.5000, 2010) spreadsheet database, with automatic checking for consistency and amplitude. The descriptive statistical analysis was conducted with the assistance of the SPSS computer program (SPSS Inc., Chicago, IL, USA, Release 16.0.2, 2008). Categorical variables were translated using absolute and relative frequency, while continuous variables were illustrated by mean ± standard deviation after verification of the normality of the data using the Kolmogorov-Smirnoff test.

The level of the reproducibility of the MCTQ was calculated using the Student t-test for repeated measures, which was used to compare the values of the test and the re-test. Possible correlations between these data were also verified using the Pearson linear correlation coefficient (r) and by the coefficient of determination (r²) which is a measure of the proportion of variability of one variable that is explained by the variability of another. The Pearson linear correlation coefficient varied between -1 and 1. The value indicates the magnitude of correlation and the sign indicates the direction (negative or positive). The closer to 1, the stronger the level of linear association between the variables. On the other hand, the closer to zero, the lower the level of association or its insignificance. ²³ The coefficient of determination is a measure of the proportion of variability of a variable that is explained by the variability of another.

To evaluate the measure of reliability, the internal consistency was used, by means of the standardized Cronbach coefficient. Cronbach’s Alpha is an index used to measure the reliability of the internal consistency of a scale, verifying the magnitude at which the items of an instrument are correlated. Accordingly, it consists of the means of the correlations between the items that form part of an instrument. The minimum acceptable value for Alpha is 0.70. Usually values of Alpha between 0.80 and 0.90 are preferred. Values above 0.90 are considered redundant, meaning that different items are measuring exactly the same element of a construct. ^24,25

The degree of concordance between the MCTQ responses, validating the construct, was evaluated using the Kappa index. The Kappa is a measure of the intra-observer and inter-observer concordance and measures the degree of concordance as well as what would be expected from chance alone and generally varies from 0 to 1 (although negative numbers are possible), where “0” represents there being no concordance other than pure chance and “1” represents perfect concordance. ²⁶ The values were distributed in five concordance categories for the reliability estimates, from the results found for Kappa: nearly perfect (≥0.81); substantial (0.61 to 0.80); moderate (0.41 to 0.60); fair (0.21 to 0.40); weak (0.01 to 0.20) and poor (0.00). ²⁷ All of the statistical analyses are two-tale, p values when calculated are exact, and the significance level adopted was 5%.

RESULTS

A total of 835 questionnaires were completed, being 427 tests and 408 re-tests, by 246 students in the fifth and sixth years of the Medical courses at four higher education institutions. Nineteen evaluations were excluded for being incorrectly completed by the student or because a re-test had not been completed. Accordingly, there were 816 questionnaires included (408 tests and 408 re-tests) in the present analysis. The students evaluated 84 preceptors in four scenarios of compulsory course internships: pediatrics; gynecology and obstetrics; medical clinic; surgical clinic. Each preceptor was evaluated at least twice. Each student evaluated up to three preceptors for each internship.

As the objective in question was to validate the instrument itself and the student was not required to self-identify, there was no investigation of the data related to the study population (age, sex, etc.). In the study underlying this research, the questionnaires were completed anonymously. ¹⁵

The students spent on average six to seven minutes completing the MCTQ. This time interval was close to that used in the transcultural validation step ¹⁶ and in the original study involving Dutch students. ¹⁵

Reliability

The analysis conducted involved comparing the responses in the test and the re-test. In this study, the Cronbach Alpha Coefficient was greater than 0.70 for all of the dimensions of the questionnaire (varying between 0.77 and 0.94), demonstrating good internal consistency in the proposed dimensions, resulting in an instrument with measured reliability. ( Table 1 and 2 ).

TABLE 1
Mean score (Scale 1: = fully disagree, 5 = fully agree), corresponding standard deviation (SD) for each item of the Maastricht Clinical Teaching Questionnaire and overall judgment (scale 1 to 10) of the test.

Test: questionnaire domains and items	Mean (1–5)	SD	Alpha coefficient
Modeling 1.Consistently demonstrated how to perform clinical skills. 2.Created sufficient opportunities for me to observe him/her. 3.Served as a role model as to the kind of doctor I would like to become.	4.4 4.0 4.1	0.95 1.04 1.11	0.85
Coaching 4.Gave useful feedback during or immediately after direct observation of my patient encounters. 5.Adjusted his/her teaching activities to my level of experience. 6.Offered me sufficient opportunities to perform activities independently.	4.2 4.2 4.3	1.07 0.96 0.93	0.77
Articulation 7.Asked me to provide a rationale for my actions. 8.Asked me questions aimed at increasing my understanding. 9.Stimulated me to explore my strengths and weaknesses.	4.4 4.4 3.9	0.90 0.90 1.09	0.85
Exploration 10.Encouraged me to formulate learning goals. 11.Encouraged me to pursue my learning goals.	3.9 4.0	1.09 1.06	0.91
Safe learning environment 12.Created a safe learning environment. 13.Was genuinely interested in me as a student. 14.Showed that he/she respected me.	4.2 4.1 4.5	1.14 1.06 0.94	0.88
Overall judgment of clinical teaching (1 – 10)	8.5	1.64

TABLE 2
Mean score (Scale 1: = fully disagree, 5: = fully agree), corresponding standard deviation (SD) for each item of the Maastricht Clinical Teaching Questionnaire and overall judgment (scale 1 to 10) of the re-test.

Re-test: questionnaire domains and items	Mean (1–5)	SD	Alpha coefficient
Modeling 1.Consistently demonstrated how to perform clinical skills. 2.Created sufficient opportunities for me to observe him/her. 3.Served as a role model as to the kind of doctor I would like to become.	4.3 4.0 4.1	0.96 1.03 1.13	0.88
Coaching 4.Gave useful feedback during or immediately after direct observation of my patient encounters. 5.Adjusted his/her teaching activities to my level of experience. 6.Offered me sufficient opportunities to perform activities independently.	4.1 4.2 4.3	1.05 1.00 0.90	0.82
Articulation 7.Asked me to provide a rationale for my actions. 8.Asked me questions aimed at increasing my understanding. 9.Stimulated me to explore my strengths and weaknesses.	4.3 4.3 3.9	0.91 0.97 1.10	0.88
Exploration 10.Encouraged me to formulate learning goals. 11.Encouraged me to pursue my learning goals.	3.9 4.0	1.12 1.08	0.94
Safe learning environment 12.Created a safe learning environment. 13.Was genuinely interested in me as a student. 14.Showed that he/she respected me.	4.1 4.1 4.5	1.19 1.08 0.90	0.89
Overall judgment of clinical teaching (1 – 10)	8.5	1.64

When the MCTQ questions are analyzed individually ( Table 3 ), the results also confirm the excellent reliability of the instrument, with values of 0.84 or above.

TABLE 3
Individual analysis of the relationship between each response in the MCTQ at the two times the preceptors were evaluated.

Questions	Test (n=408)		Re-test (n=408)		p -value	Cronbach’s Alpha	Kappa	r*	r2
	Mean (1–5)	SD	Mean (1–5)	SD
1	4.4	0.95	4.3	0.96	0.466	0.91	0.645	0.837	0.701
2	4.0	1.04	4.0	1.03	0.732	0.92	0.640	0.843	0.711
3	4.1	1.11	4.1	1.13	0.192	0.95	0.710	0.904	0.817
4	4.2	1.07	4.1	1.05	0.030	0.90	0.570	0.821	0.674
5	4.2	0.96	4.2	1.00	0.337	0.89	0.541	0.801	0.642
6	4.3	0.93	4.3	0.90	0.698	0.86	0.527	0.758	0.575
7	4.4	0.90	4.3	0.91	0.022	0.84	0.561	0.726	0.527
8	4.4	0.90	4.3	0.97	0.001	0.89	0.575	0.810	0.656
9	3.9	1.09	3.9	1.10	0.726	0.88	0.537	0.791	0.626
10	3.9	1.09	3.9	1.12	0.640	0.91	0.624	0.835	0.697
11	4.0	1.06	4.0	1.08	0.452	0.90	0.609	0.811	0.658
12	4.2	1.14	4.1	1.19	0.018	0.93	0.601	0.866	0.750
13	4.1	1.06	4.1	1.08	0.116	0.93	0.670	0.875	0.766
14	4.5	0.94	4.5	0.90	0.655	0.94	0.656	0.885	0.783

• ²

Construct validity

The construct validity of the instrument was tested item by item. The concordance indexes are shown in table 3 . The Kappa index between the items of the test and the re-test varied from 0.527 in questions six: “offered me sufficient opportunities to perform activities independently” to 0.710 in question three: “served as a role model as to the kind of doctor I would like to become”, demonstrating a moderate to substantial degree of concordance for the instrument.

Reproducibility

The reproducibility of the instrument was also tested item by item, comparing two paired samples, being test/re-test, with the same subjects at two distinct times. The minimum interval between the test and the re-test was one week and the maximum was three weeks. The aim was for the period between the evaluations not to be excessively long, which could result in variations in the responses for the items analyzed. On the other hand, the interval was not too short, to prevent memorizing of the responses (learning effect). All of the correlation coefficient values found were significant and of good magnitude (≥0.72). The results obtained show a strong positive linear correlation for 13 questions, with a very strong correlation, almost perfect, of (+) 0.90, found for question three.

The coefficient of determination (r²), the square of the Pearson linear correlation coefficient (r), was high (>0.60) for 12 of the 14 questions analyzed, a fact that shows that the test score determines or explains that of the re-test in more than 60% of times for these questions. That is, a high score in the test determines a high score in the re-test, while a low score in the test determines a low score when the questionnaire is reapplied. In contrast, question seven showed statistically significant difference in the pre and post values, having obtained a lower correlation value between the two times. In addition, questions 04, 08 and 12 also showed statistically different means between the two times of application.

DISCUSSION

Simultaneously with the changes that are being implemented in health degree course curricula, it is critical to expand investment in the training of teachers and preceptors, as well as to create new tools to evaluate the student activity as a fundamental element in this teaching-learning process.

In this regard, an instrument that evaluates the performance of the preceptor from data supplied by their students is of great importance for improving teaching practices. This is particularly evident in the specific case of Medical courses, as in the majority of teaching institutions there is no guidance given with regard to training future medical teachers.

There was excellent acceptance of the MCTQ on the part of the students during this verification and reliability phase, just as during the transcultural adaptation process. There was consensus in the view that the MCTQ is an instrument that can contribute to improving mentoring and the medical course.

Following the validation steps for the questionnaire, a strong positive linear correlation ²⁰ was found between the items of the questionnaire analyzed, equal or above 0.72, as well as a high degree of concordance between them. However, the following considerations have to be made: with questions 4: “gave useful feedback during or immediately after direct observation of my patient encounters”; 7: “asked me to provide a rationale for my actions”; 8: “asked me questions aimed at increasing my understanding”; and 12: “created a safe learning environment” there were statistically significant differences confirmed between the results found at the two times of the evaluation ( p <0.05). However, when the values obtained were analyzed for these items individually, it was noted that the variation between the scores in the test and re-test are of small magnitude, corresponding to 0.1 between the means on a scale that ranges from one to five, as in question 4, for example, which varied between 4.2 (test) and 4.1 (re-test).

A hypothesis for this small difference found in the above-mentioned questions is that it could have arisen from the variability of the time interval between the two times at which the questionnaire was applied. Hence, an interval of less than three weeks may be ideal for employing the instrument. It could also be justified to better familiarize students with the questionnaire or for them to have greater contact with the preceptor evaluated. Certainly, the student’s judgment of the pedagogical practice adopted by the preceptor could be influenced by more prolonged contact. Future studies could investigate these hypotheses.

Next, attention is called to the fact that question seven presented the smallest values for the coefficients for linear correlation, determination and Cronbach’s Alpha. This question also presented statistical difference for the means at the two times of evaluation, from which three situations could be inferred: 1. that it is difficult to understand; 2. the translation may be imprecise; 3. the preceptor evaluated really did not require logical reasoning on the part of the student.

In the exploration domain, it was interesting to see a high Cronbach’s Alpha coefficient (>0.90) in both the test and the re-test. This domain consists of items 10: “encouraged me to formulate learning goals” and 11: “encouraged me to pursue my learning goals”. This finding suggests redundancy in the information for such questions, that is, these questions are possibly measuring the same element of the construct. ²⁴ In a similar manner, a value above 0.90 was also found for the “exploration” domain in the original study. ¹⁵ Given the similar mathematical result, it is understood that it was not the validation process that was responsible for this possible redundancy.

On the other hand, although the previous study also found a Cronbach’s Alpha coefficient above 0.90 for the domain “safe learning environment”, in this study the value obtained was between 0.88 and 0.89 for the test and the re-test, respectively, showing a statistically acceptable Cronbach’s Alpha coefficient. Given that values greater than 0.90 for this indicator suggest redundancy between the items, the orthographical and grammatical modifications in the transcultural validation or a possible greater understanding by the students evaluated could favor internal consistency in the construct in the Portuguese language.

Individually evaluating the score means attributed to the questions in the test and re-test, on a scale of 1 to 5, shows that the largest mean (4.5) was attributed to item 14, “showed that he/she respected me”. This demonstrates that there was generally a good relationship between the preceptor and the student in the study population, independently of the preceptor not having been evaluated in the other items. On the other hand, questions 9, “stimulated me to explore my strengths and weaknesses” and 10, “encouraged me to formulate learning goals”, obtained the smallest means (3.9). In the sample analyzed, these findings suggest a greater need to motivate the preceptor in relation to their student.

As a limitation of this study, it can be cited that only students registered in four medical schools in the Northeast of Brazil were included, with no students on other course and in other regions of the country. However, it is important to stress that the original study was restricted to students at a single Dutch teaching institution. ¹⁵ Hence, the fact that students at more than one higher education institution were included could also be considered to be a strong point of the study.

Another limiting factor in this research that can be cited arises from having no distinction between the scores given by the students in the fifth and sixth year. Certainly, student perception changes depending on the extent of learning acquired during the course. The student who is starting the internship requires more attention from the preceptor, than the medical resident and students in more advanced periods. This is a frequent obstacle in clinical teaching evaluations, ²⁸ a source for new studies.

It is also stressed that the students placed in the basic health care units and the associated preceptors did not participate in the study. This was because of the difficulties in applying the questionnaires (test/re-test) in these internships. These take place in numerous basic health units spread throughout the capital of the state of Pernambuco and the São Francisco Valley, complicating the collection of data and follow-up of the students during the study. However, the majority of them participated in the research in the subsequent weeks when the clinical scenarios they were in changed. It is also important to highlight that the Dutch study was conducted only in the internships taking place within the hospital environment. ¹⁵

The fact that the adapted MCTQ has been applied in this research only in a population formed by medical students does not prevent it being used for other courses in which there are learning scenarios and teacher-student relationships similar to those occurring on the medical course. In the previous work for the transcultural adaptation of the questionnaire, it was answered by students from other courses in the health area and was well accepted (pre-test). ¹⁶ In addition, it is a fact that previous studies using this instrument have been valid when applied to a veterinary medical course, ²⁹ as well as in post-graduate courses. ³⁰

Hence, the transculturally adapted MCTQ, validated for use in the Portuguese language, has been shown to be an important tool that could provide practical results helping preceptors both for medical courses and for other courses in the health area. The fact that in Brazil there is no validated instrument with similar characteristics hinders comparisons with the adapted version of the MCTQ, while making it innovative.

Subsequently, from the use of the adapted version, it will be possible to conduct new studies evaluating the impact provided by this tool as an instrument for promoting improvements in the pedagogical aspect of degree and post-graduate courses, notably those taking place in the health sciences field.

CONCLUSION

From the results presented in this study, it can be concluded that the Portuguese version of the MCTQ is reliable and valid for use in the Portuguese language with students on courses in the health area.

REFERÊNCIAS

• ¹
  Franco FM , Montes MAA , da Silva AR . Visão discente do papel da preceptoria médica na formação dos alunos de medicina . Alexandria: Rev Educ Cienc Tecnol . 2013 ; 6 ( 2 ): 229 - 49 .
• ²
  Troncon LEA , Bollela VR , Borges MC , Rodrigues MLV . A formação e o desenvolvimento docente para os cursos das profissões da saúde: muito mais que o domínio de conteúdos . Medicina ( Ribeirão Preto ). 2014 ; 47 ( 3 ): 245 - 8 .
• ³
  Freitas DA , Santos EMS , Lima LVS , Miranda LN , Vasconcelos EL , Nagliate PC . Teachers’ knowledge about teaching-learning process and its importance for professional education in health . Interface ( Botucatu ). 2016 ; 20 ( 57 ): 437 - 48 .
• ⁴
  Flexner A . Medical e ducation in the United States and Canada. From the Carnegie Foundation for the Advancement of Teaching, Bulletin Number Four, 1910. (Bulletin, 4) . Bull World Health Organ . 2002 ; 80 ( 7 ): 594 - 602 .
• ⁵
  Moura DTD , Foltran RS , Fraiz IC , Novak EM . Articulação entre os ciclos básico e profissionalizante: percepção dos alunos da UFPR . RevBrasEduc Med . 2018 ; 42 ( 1 ): 226 - 36 .
• ⁶
  PagliosaI FL , Da Ros MA . O relatório Flexner: para o bem e para o mal . RevBrasEduc Med . 2008 ; 32 ( 4 ): 492 - 9 .
• ⁷
  Brasil . Ministério da Educação . Conselho Nacional de Educação . Diretrizes Curriculares Nacionais para o Curso de Graduação em Medicina. Parecer CNE/CES n° 116/2014, publicado no DOU de 6 de junho de 2014 .
• ⁸
  Brasil . Ministério da Saúde . Programa mais médicos . Disponível em: http://maismedicos.gov.br/ . Acessado em 31 de outubro de 2015 .
  » http://maismedicos.gov.br/
• ⁹
  Miller GE . The assessment of clinical skills/competence/performance . Acad Med . 1990 ; 65 ( 9 Suppl ): S63 - 7 .
• ¹⁰
  Cruess RL , Cruess SR , Steinert Y . Amending Miller’s pyramid to include professional identity formation . Acad Med . 2016 ; 91 ( 2 ): 180 - 5 .
• ¹¹
  Benbassat J . Role modeling in Medical Education: the Importance of a reflective imitation . Acad Med . 2014 ; 89 ( 4 ): 550 - 554 .
• ¹²
  Passi V,Johnson N . The impact of positive doctor role modeling . Med Teach . 2016 ; 38 ( 11 ): 1139 - 45 .
• ¹³
  Boerboom TBB , Stalmeijer RE , Dolmans DHJM , Jaarsma DADC . How feedback can foster professional growth of teachers in the clinical workplace: a review of the literature . Stud EducEval . 2015 ; 46 : 47 - 52 .
• ¹⁴
  Stalmeijer RE , Dolmans DH , Wolfhagen IH , Muijtjens AM , Scherpbier AJ . The development of an instrument for evaluating clinical teachers: involving stakeholders to determine content validity . Med Teach . 2008 ; 30 ( 8 ): e272 - 7 .
• ¹⁵
  Stalmeijer RE , Dolmans DH , Wolfhagen IH , Muijtjens AM , Scherpbier AJ . The Maastricht Clinical Teaching Questionnaire (MCTQ) as a valid and reliable instrument for the evaluation of clinical teachers . Acad Med . 2010 , 85 ( 11 ): 1732 - 8 .
• ¹⁶
  Vasconcelos PT . Adaptação transcultural do Maastricht ClinicalTeachingQuestionnaire para a língua portuguesa . Recife ; 2015 . Mestrado Profissional em Educação para o Ensino na Área de Saúde[Dissertação] – Faculdade Pernambucana de Saúde – FPS .
• ¹⁷
  Souza AC , Alexandre NMC , Guirardello EB . Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade . EpidemiolServ Saúde . 2017 ; 26 ( 3 ): 649 - 59 .
• ¹⁸
  Ramada-Rodilla JM , Pujadas CS . Adaptación cultural y validación de cuestionarios de salud: revisión y recomendaciones metodológicas . SaludPúblicaMéx . 2013 ; 55 ( 1 ): 57 - 66 .
• ¹⁹
  Guillemin F , Bombardier C , Beaton D . Cross cultural adaptation of health quality of life measures: literature review and proposed guidelines . J ClinEpidemiol . 1993 ; 46 ( 12 ): 1417 - 32 .
• ²⁰
  Duffy ME : Translating instruments into other languages: basic considerations . Clin Nurse Spec . 2006 ; 20 : 225 – 226 .
• ²¹
  Ramada-Rodilla JM , Pujadas CS . Adaptación cultural y validación de cuestionarios de salud: revisión y recomendaciones metodológicas . Saludpública de méxico . 2013 ; 55 ( 1 ): 57 - 66 .
• ²²
  Muñiz J , Elosua P , Hambleton RK . Directrices para la traducción y adaptación de los tests: segunda edición . Psicothema . 2013 ; 25 ( 2 ): 151 - 157 .
• ²³
  Mukaka MM . A guide to appropriate use of Correlation coefficient in medical research . Malawi Med J . 2012 ; 24 ( 3 ): 69 - 71 .
• ²⁴
  Streiner DL . Being inconsistent about consistency: when coefficient alpha does and doesn’t matter . J Pers Assess . 2003 ; 80 ( 3 ): 217 - 22 .
• ²⁵
  Tavakol M , Dennick R . Making sense of Cronbach’s alpha . Int J Med Educ . 2011 ; 2 : 53 - 5 .
• ²⁶
  Pinto JS , Lopes JM , Oliveira JV , Amaro JP , Costa LD . Métodos para estimação de reprodutibilidade de medidas [ online ]. Porto – Portugal : Faculdade de Medicina do Porto ; 2008 . Disponível em: http://users.med.up.pt/joakim/intromed/estatisticakappa.htm . Acessado em 31 de outubro de 2015 .
  » http://users.med.up.pt/joakim/intromed/estatisticakappa.htm
• ²⁷
  Tan CL , Hassali MA , Saleem F , Shafie AA , Aljadhey H , Gan VB . Development, test-retest reliability and validity of the Pharmacy Value-Added Services Questionnaire (PVASQ) . Pharm Pract ( Granada ). 2015 ; 13 ( 3 ): 598 .
• ²⁸
  Beckman TJ , Mandrekar JN . The interpersonal, cognitive and efficiency domains of clinical teaching: construct validity of a multidimensional scale . Med Educ . 2005 ; 39 ( 12 ): 1221 - 29 .
• ²⁹
  Boerboom TBB , Mainhard T , Dolmans DHJM , Scherpbier AJJA , van Beukelen P , Jaarsma ADC . Evaluating clinical teachers with the Maastricht clinical teaching questionnaire: how much ‘teacher’ is in student ratings? Med Teach . 2012 ; 34 ( 4 ): 320 - 6 .
• ³⁰
  Giannasi S , Durante E , Ladenheim R , Kecskes R , Vázquez F , Stalmeijer R , et al . Transcultural validation in Spanish of the Maastricht Clinical Teaching Questionnaire (MCTQ) as a valid and reliable instrument for the evaluation of clinical teachers . Milano, Italy : AMEE ; 2014 .

Publication Dates

• Publication in this collection
  Apr-Jun 2019

History

• Received
  15 Sept 2018
• Accepted
  8 Oct 2018