Clinical skills assessment and feedback in pediatric residency

Método: Trata-se de uma pesquisa-ação educacional (pesquisa-ensino) realizada com docentes e preceptores da residência médica em pediatria de um hospital universitário. As etapas consistiram em: 1. aplicação de questionário sobre o perfil dos participantes e os métodos avaliativos utilizados com os residentes; 2. intervenção com a realização de um workshop sobre avaliação de habilidades clínicas e feedback; 3. avaliação imediata, após o workshop, com aplicação de outro questionário, elaborado com base no nível 1 do método Kirkpatrick. Utilizaram-se a análise estatística simples, para os dados objetivos, e a análise de conteúdo, segundo recomendações de Malheiros e Bardin, para a parte qualitativa.


INTRODUCTION
Medical residency is the specialization par excellence in the training of physicians, and the program is responsible for ensuring that recently graduated doctors reach the desired level of competence 1 . As it is in-service training, assessment, in this context, goes beyond the cognitive evaluation, constituting a daily challenge for the teacher and the preceptor.
An effective evaluation process requires, in addition to teacher training, a system that combines several types of evaluation, aiming to reach all elements of learning (knowledge, skills and attitudes), in addition to ensuring the validity and reliability of the utilized methods 2 .
Considering the complexity of the teacher's role in medical education, the development and implementation of teacher development programs, activities that seek to improve the knowledge and skills of health professionals as teachers, should be considered a permanent process 3,4 . However, according to Steinert 4  [...] Art. 13. In the periodic evaluation of the resident physician, the modalities of written, oral, practical or performance tests by an attitude scale will be used, which include attributes such as: ethical behavior, relationship with the health team and with the patient, interest in activities and others at the discretion of the institution's COREME. §1. The minimum frequency of evaluations will be every three months. §2º. At the institution's discretion, a monograph and/or presentation or publication of a scientific article may be required at the end of the training. §3º The criteria and the results of each evaluation must be made known to the resident doctor.
As for the evaluation, a conceptual model proposed by Miller, known for several decades, known as the Miller's pyramid, demonstrated to teachers that, regarding professional development, the evaluation cannot be restricted to theoretical knowledge, as it is necessary for the student to know how to apply this knowledge, perform it, in a practical way, in simulated environments and, finally, apply it in real life 8 .
Miller's pyramid aligns its strata with the educational objectives and evaluation methods aimed at the types of skills and competences whose domain one wants to know, rising from the theoretical knowledge contained at the base -"to know" and "to know how" -to "to show how" and "to perform". The pyramid apex corresponds to the evaluation of professionals in their work environment 9 .
Based on these dimensions and the resident doctor's degree of learning, the observations made by teachers and preceptors should be directed, in addition to the cognitive one, to the performance evaluations, considering the clinical and psychomotor skills, the interaction with the patient, the management of information, the capacity for judgment, synthesis and decision, as well as the preservation of ethical attitudes 1 .
Most clinical skills evaluation methods have, as a basic principle, the direct observation of the resident's performance in clinical tasks, in a real or simulated environment. In this sense, the performance of feedback should be allowed, preferably an immediate (formative) one, which consists in describing and discussing with the residents their performance related to a given activity 10 .
A well-developed and periodic evaluation system, with continuous feedback, is an effective tool to improve the performance of the future specialists and guarantee their qualification, a goal of indisputable importance in the training process 2 . For that purpose, the resident's evaluation needs the systematization and institutionalization related to how to evaluate, in addition to teacher training for this important aspect of the teaching-learning process.
This research proposed to answer the following question: how are residents being evaluated regarding the skills acquired in the pediatric medical residency program at Hospital Universitário Professor Alberto Antunes? Hence, the objective was to analyze the system used to evaluate the pediatric resident doctor of a university hospital, aiming to promote teacher training in evaluation methods.

METHODOLOGICAL TRAJECTORY
An educational action research was created aiming to identify gaps in pedagogical practice and cause changes in educational habits, considering its potential as an investigative praxis, in the resident's evaluation process through an intentional sample with teachers and preceptors of the aforementioned residency program in pediatrics.
In the educational field, the action research (researchteaching) consists of an investigation about the practice itself and implies the awareness by the participants, allowing them to be involved in all phases of the methodological trajectory 11,12 . Its use in the educational field allows the researcher teacher to identify a problem in their pedagogical activity and, through research, create the conditions to transform it, aiming to favor the personal and professional growth of the researchers and the involved participants 11,12 .
The action research follows a cycle in which practice is improved in the movement between acting in the field of the problem and investigating about it. The cycle includes the identification of the problem and data production on the effects of a change in practice during the intervention, before and after its implementation, using pre-and post-methods to monitor the effects caused by the change 13 ( Figure 1).
According to Malheiros 14 , this methodology is very useful in the educational field, because it allows studies on changes in curriculum, teaching-learning models, evaluation methods, among other aspects.
In the research development, different procedures were used for data collection, divided into three stages, observing and complying with the action research methodology, aiming to understand the reality of the resident physician's evaluation and to propose action strategies for its improvement. where the pediatric residents from Hupaa work in the urgency/ emergency and infectious diseases sectors, respectively. In this phase, one researcher teacher was excluded from the study, and the others agreed to participate in the discussion and to answer the questionnaire.
The first semi-structured questionnaire was applied to identify the deficiencies in the residents' evaluation process, consisting of questions related to the participants' sociodemographic data, specific data on the teachers' training in assessment, the evaluation methods (cognitive tests or assessment of clinical skills) that they used with the resident physicians, the factual knowledge of performance evaluation methods in real and simulated environments, the perspective of teachers and preceptors regarding the evaluation process, as well as the intention to participate in a teaching development workshop about an evaluation method.
The participants' answers to the open questions were organized based on the ideas that emerged from the guiding questions, when the pre-analysis was carried out through further reading, observing the emergence of categories that were not previously created. A matrix was created, and all statements were transcribed in full. The participants were coded by letters and numbers, following the order of analysis of the questionnaires -teacher (T) and preceptor (P).
Matrices were created, which stored the explicit or implicit ideas, the creation of the categories and the registration units that associate the statements to the topic to explain, in the text, how the result was achieved. The focal points and recording units were interpreted, and the synthesis for each focus was developed 14 .
Simple statistical analysis was used for objective data and content analysis, according to the recommendations by    As for the evaluation methods, 81% (17/21) of the participants reported using more than one, for summation purposes, to obtain a more comprehensive and reliable evaluation. According to Norcini et al. 19 , the structure for a good evaluation consists of an organized combination of methods, to constitute an evaluation system; however, none of the teachers/preceptors uses a systematized evaluation of the resident physician's clinical, psychomotor or affective skills, or provide feedback. Therefore, despite the diversity of evaluation methods, there is no guarantee of the absence of weaknesses 20 .
In the statements provided by teachers and preceptors, a collective concern with this inadequacy related to the evaluation methods can be observed: This corroborates the results of Zimmerman et al. 20 , where teachers declared having difficulty in evaluating and formulating tests due to the lack of theoretical basis, in addition to the lack of standardization in the medical course.
The importance of performance evaluation, at this level of training, is due to its potential to verify clinical skills The evaluation improvement of teachers and preceptors ensures the quality of the evaluation and the teaching-learning process, since the evaluation allows the review of educational planning and adjustments in their teaching practice 20 .
One of the statements refers to the need for feedback, as stated by T3: "Continuous assessment, in daily life, is essential for the learning process, always with feedback for its strengthening". In this statement, it can be observed that the teacher synthesizes the entire process of one evaluation with a formative purpose.
Feedback is the substrate of a formative assessment and an effective tool to improve student performance, especially when it is performed immediately, after the clinical task 10 . Therefore, it must take place in a dialogical way, with the student playing an important role in the evaluation of their own performance 24 .
In medical education, feedback is as essential for educators to promote learning as it is for students, as it provides information about their work and quality, aiming to generate improvements 25 .
The results showed that the participants' concern about the residents' assessment methods was a collective one, considering this as the first step towards institutionalizing changes within the teaching-learning scenario.

Stage 2 -Workshop: integrative and interactive dynamics
After identifying the problem, evidenced by the absence of systematization, institutionalization and teacher training regarding evaluation methods for residents in pediatrics, an intervention was planned and implemented as a teaching  The butterfly effect 28 caused by the intervention (workshop), in the educational action research proposal, and through the evaluation method itself, has also become a teaching-learning environment for students from other levels of education. It is expected that this effect, a metaphor used in science, described by Edward Lorenz in 1972 28 , caused by a small change at the beginning of an event, can have positive future consequences and that it will unfold into new interventions in evaluation methods at the institution, for all undergraduate and graduate levels. In this sense, it is in education and in the reflections that come from it that the transformative potential of changes ensues.
The limitations of the study include the difficulty in aggregating all the participants involved (teachers, preceptors and actors), due to the conciliation of time, place and work, in addition to addressing only one method of evaluation, due to the organization's logistics and practicality.

FINAL CONSIDERATIONS
The action research carried out with the teachers and preceptors who work with the medical residency program allowed, mainly, to identify limitations in the evaluation and Although reflections on the inadequacy of the evaluation process have been generated, the intervention (workshop) was not sufficient to positively interfere, in the short term, with the pediatric medical residency evaluation.
It is known that a well-designed evaluation process requires time, periodicity, planning and organization. Solid changes require other factors, in addition to continuing teacher training in teaching-learning; it requires dedication, commitment from the involved educators, teachers and preceptors, student awareness and a de facto institutional formalization.
Evaluating is a complex process that remains a challenge; however, promoting spaces for debates and reflections, as shown by the research, can lead to significant measures and they might evolve towards the educational objectives aimed at the physician in training.